Senior Data Engineer – AWS / Databricks / PySpark

Your tasks

We are seeking an experienced Senior Data Engineer to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.

Requirements

Key Responsibilities

• Design, develop and optimise scalable batch and streaming data pipelines using AWS, Databricks, Python, PySpark and SQL.
• Build and maintain lakehouse architectures using Delta Lake, including Bronze, Silver and Gold data layers, curated datasets and reusable data products.
• Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.
• Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.
• Develop and automate data workflows using Databricks Workflows, Lakeflow and AWS services, incorporating CI/CD, automated testing and infrastructure as code.
• Implement data governance, access controls and security measures using Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM.
• Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.
• Lead technical design and code reviews, establish engineering standards and mentor junior engineers.
• Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
• Maintain technical documentation, data models, operational procedures and knowledge transfer materials.

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
• 6–10 years of data/software engineering experience, including at least 2 years working with cloud-based data platforms.
• Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake.
• Proficiency in AWS data services such as S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS.
• Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.
• Strong understanding of ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage.
• Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably Terraform.
• Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation.
• Familiarity with cloud security, encryption, access management and data governance.
• Proven ability to provide technical guidance, conduct code reviews and mentor engineers.

Good to Have

• Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.
• Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS.
• Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts.
• Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps.
• Exposure to Singapore public-sector environments, including IM8, VAPT, data classification and cross-agency data governance.

Other

Senior Data Engineer – AWS / Databricks / PySpark