Senior Data Engineer

We are looking for a Senior Data Engineer to help drive the evolution of our Azure-based Data Platform and Lakehouse architecture .

This role is focused on Data Products development. The successful candidate will build pipelines in ADF, DAGs in Airflow and work closely with POs to refine business requirements and transform business needs into reliable Data Products in Databricks.

Daily tasks

• Implement scalable , reliable and reusable data pipelines using Azure Data Factory and Apache Airflow.
• Develop data transformations in Azure Databricks (PySpark / SQL) to create/extend data products following theMedallion Architecture layers: Bronze, Silver, Gold,
• Optimize performance, cost, and reliability of data workloads.
• Raise important questions during business maturation in order to detect risks before development stage.
• Participate in architecture reviews and technical decision-making.
• Promote engineering excellence through code reviews and design reviews.
• Collaborate with Architects, Product Owners in data product technical maturation.
• Provide support to the Run/Operations team for production pipelines and data products for complex topics.
• Troubleshoot and resolve data incidents, failures, and performance issues
• Participate in incident management, root cause analysis, and post-mortems.
• Ensure proper handover, documentation, and operational readiness of developed solutions.
• Contribute to improving runbooks, monitoring, and operational processes.

Requirements

• 6+ years of experience in data engineering
• Python (PySpark)
• SQL (advanced level)
• Hands-on experience with Azure-based Data Platform including: Databricks ,Unity Catalog , Apache Airflow, Azure Data Factory, Reporting and BI tools (Power BI, Tableau or similar)
• Solid understanding of data modelling fundamentals, main architectures and design patterns
• Data Warehouse, Data Lake and Data Lakehouse architectures
• Medallion architecture
• Metadata-driven architecture
• ETL/ELT design patterns
• Understanding of Partitioning, optimization, and performance and batch and streaming pipelines

Must have: ADF, Airflow, Databricks, Product development, Data pipelines, Azure Data Factory, Azure Databricks, Apache Airflow, PySpark, SQL, Data engineering, Python, UNITY, Power BI, Tableau, Data modelling, Design Patterns, Data warehouse, Data Lake, ETL, Communication skills