Senior Data Engineer

We are looking for an experienced senior python data engineer

with strong expertise in python development, microservices architecture, data engineering, and backend API development. The ideal candidate will have hands-on experience building scalable python-based applications and data services, designing robust REST APIs, performing complex data transformations, and developing high-performance backend frameworks to support enterprise data-driven applications.

This role requires strong proficiency in python web frameworks, asynchronous programming, microservices architecture, ETL processes, database management, and modern software engineering best practices. Exposure to big data technologies, DevOps practices, and cloud-native architectures will be highly advantageous.

Requirements

Key responsibilities

Design, develop, and maintain scalable python-based web frameworks and backend services.

Develop robust RESTful APIs using python frameworks such as FastAPI, Flask, or Django.

Build and maintain microservices-based applications supporting enterprise data platforms.

Design and implement data transformation pipelines for data ingestion, processing, and enrichment.

Develop backend systems to efficiently serve datasets through APIs and other interfaces.

Collaborate with cross-functional teams to deliver scalable and secure data-driven applications.

Design and maintain relational and NoSQL database schemas.

Develop efficient ETL (extract, transform, load) processes for enterprise data requirements.

Optimize application and database performance to ensure scalability and reliability.

Implement asynchronous programming techniques using asyncio and optimize concurrent connections and I/O operations.

Ensure data integrity, consistency, and quality across data transformation pipelines.

Participate in code reviews and contribute to software engineering best practices.

Troubleshoot, debug, and resolve production issues across applications and data pipelines.

Collaborate closely with stakeholders and engineering teams throughout the SDLC.

Required technical skills

Core python development

Strong hands-on expertise in core python.

Experience building enterprise-grade python applications.

Strong understanding of:

Python programming concepts

Object-oriented programming

Asynchronous programming (asyncio)

Concurrent connections handling

I/O optimization techniques

Clean code and software design principles

Python web frameworks

Hands-on experience with one or more of the following:

FastAPI

Flask

Django

Pyramid

Strong understanding of:

RESTful API development

Routing

Authentication mechanisms

Testing frameworks

Database integrations

Framework architecture and scalability

Microservices architecture

Strong experience with:

Microservices-based application design

Service-oriented architectures

API integrations

Distributed systems

Enterprise backend services

Knowledge of:

API gateways

Service communication patterns

Scalable backend architectures

Security & API management

Experience implementing:

OAuth

JWT authentication

API security best practices

Authorization mechanisms

Encryption standards

Secure API development practices

Data engineering & data transformations

Strong hands-on experience with:

Data transformation pipelines

ETL processes

Data cleansing and enrichment

Data aggregation techniques

Data quality and integrity management

Experience in:

Dataset preparation

Data processing workflows

Enterprise data platforms

Database management

Experience working with:

Relational databases

PostgreSQL

MySQL

SQL Server (preferred)

NoSQL databases

MongoDB

Redis (good to have)

Strong knowledge of:

Database schema design

Performance optimization

Transactions management

Efficient data retrieval techniques

ORM frameworks

Hands-on experience with:

SQLAlchemy

Django ORM

Ability to perform:

CRUD operations

Query optimization

Transaction handling

Data modelling

DevOps & CI/CD

Experience with:

CI/CD pipeline development and maintenance

Git-based workflows

Version control best practices

Branching strategies

Code reviews

Production deployments

Big data technologies (preferred)

Exposure to:

Apache Spark

Hadoop

Apache Kafka

Understanding of:

Data warehousing concepts

Large-scale data processing architectures

Scripting skills

Strong proficiency in:

SQL

Shell scripting

Python scripting

Cloud & containerization (good to have)

Exposure to:

AWS

Microsoft Azure

Google Cloud Platform (GCP)

Docker

Kubernetes

Data governance (good to have)

Knowledge of:

Data governance principles

Data security best practices

Compliance frameworks

Enterprise data management practices

Preferred additional skills

Experience with R programming language.

Experience working on enterprise-scale data engineering initiatives.

Exposure to cloud-native architectures and containerized deployments.

Required competencies

Strong analytical and problem-solving skills.

Excellent communication and stakeholder management capabilities.

Strong debugging and troubleshooting skills.

Ability to work effectively in agile and collaborative environments.

Strong ownership mindset and attention to detail.

Ability to adapt quickly to evolving business and technical requirements.

Strong documentation and technical communication skills.