Data Engineer / Senior Data Engineer

Requirements

Essential (Must-Have)

• Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query optimisation, debugging and automated testing.
• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.
• Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, Event Bridge, CloudWatch, Redshift and MSK/Kinesis.
• Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or domain data modelling and reusable curated datasets.
• Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery.
• Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of production pipelines.
• Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform, and environment promotion controls.
• Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.
• Working knowledge of cloud security, least-privilege access, network boundaries, encryption, secret management, data classification and audit logging.
• Ability to review technical designs and code, establish engineering standards, and guide less-experienced engineers.

Preferred (Advantageous)

• Experience with Databricks Unity Catalog, Lakeflow or Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.
• Experience with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis, AWS DMS or equivalent ingestion and orchestration technologies.
• Experience designing data products, data contracts, semantic or context layers, master/reference data patterns or data mesh-aligned operating models.
• Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines on Databricks and AWS.
• Experience in Singapore public-sector environments, including IM8-aligned controls, data classification, VAPT, system assurance or governed cross-agency data sharing.
• Relevant certifications such as Databricks Certified Data Engineer Professional, AWS Certified Data Engineer –Associate or AWS Certified Solutions Architect – Associate/Professional.