Senior Data Engineer

Your tasks

building, productionising and operating trusted data pipelines and reusable data products on the data platform using AWS and Databricks.

Requirements

Experience: Typically 6–10 years of relevant data or software engineering experience, including hands-on design and

operation of production data pipelines and at least 2 years working with cloud-based data platforms.

• Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query

optimisation, debugging and automated testing.

• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming

data engineering.

• Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda,

Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis.

• Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or

domain data modelling and reusable curated datasets.

• Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling,

backfills, replay and recovery.

• Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated

parts of production pipelines.

• Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as

Other

Work location- Paya Lebar The Senior Data Engineer sits within the DSAD Platform and Architecture Team and is responsible for designing,

Terraform, and environment promotion controls.