Data Engineer

Key Responsibilities

• Design, develop, and maintain scalable data pipelines to ingest, transform, and process data from internal and external sources using Databricks and Medallion Architecture (Bronze, Silver, and Gold).
• Build and manage data ingestion, storage, and orchestration workflows across cloud environments, preferably using Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), and Delta Lake.
• Develop and maintain production-ready ETL processes using PySpark, Python, and SQL, ensuring data quality, reliability, and efficient processing.
• Optimise data pipelines, Spark workloads, and storage solutions to improve performance, scalability, and cost efficiency.
• Design and develop reusable, metadata-driven engineering frameworks to standardise data ingestion, transformation, error handling, logging, and data quality processes.
• Implement and maintain CI/CD pipelines, version control, and automated deployment processes using Azure DevOps and Git.
• Collaborate with technical teams and stakeholders to understand data requirements, address engineering challenges, and continuously improve data platform capabilities.

Is this you?

• Bachelor's degree in Computer Science, Information Technology, Information Systems, Software Engineering, or a related technical field.
• At least 3 years of relevant experience in Data Engineering, particularly in developing and maintaining production data pipelines.
• Strong, hands-on experience with Databricks, including Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog.
• Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP, with Azure experience being highly preferred.
• Familiarity with cloud-based data services and ETL tools, particularly Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS).
• Strong programming skills in Python and SQL, with the ability to develop, troubleshoot, and optimise data processing solutions.
• Experience with Git and CI/CD practices, including version control, branching strategies, and automated deployment of data solutions.
• Good understanding of data modelling, ETL/ELT processes, and Lakehouse architecture, including Medallion Architecture.
• Experience designing reusable, metadata-driven data engineering frameworks, with an understanding of modular development, testing, logging, and error handling, is a plus.
• Familiarity with Databricks Lakeflow, including Declarative Pipelines, Jobs, or Connect, is an advantage.
• Exposure to Lakebase, Databricks Apps, or Genie (AI/BI), or an interest in exploring emerging Databricks technologies, is a plus.
• Strong analytical and problem-solving skills, attention to detail, and the ability to collaborate effectively with technical teams and stakeholders.