Data Engineer - Data Lakehouse Architecture (Public Sector)

1-year contract, renewable Government project Hybrid work arrangement

Role Overview

The Data Engineers will design, build and deploy a scalable and reliable data lakehouse platform on AWS. Their responsibilities will include developing end-to-end data pipelines, establishing data quality frameworks and delivering production-ready solutions to support the organisation's data infrastructure. The Senior Consultant will additionally lead complex engineering activities and provide technical guidance and implementation oversight.

Key Responsibilities

Data Pipelines and Architecture

• Design and implement scalable ETL/ELT pipelines using AWS Glue, Step Functions, Lambda and S3.
• Build data lakehouse solutions using Apache Iceberg or Amazon S3 Tables, incorporating schema evolution, partition evolution and ACID transactions.
• Optimise pipelines for performance, cost and reliability.

Data Quality and Governance

• Develop and maintain automated data quality validation frameworks.
• Establish metrics and monitoring to ensure data accuracy, completeness and consistency.
• Implement data governance controls and ensure compliance with applicable standards.

Application Development and Deployment

• Write production-quality code and deploy solutions on AWS.
• Implement CI/CD pipelines and infrastructure as code using tools such as Terraform or CloudFormation to support repeatable and auditable deployments.

Collaboration and Handover

• Work with cross-functional teams and communicate technical designs clearly to technical and non-technical stakeholders.
• Prepare technical documentation and support handover to the Day 2 operations team.

Qualifications and Experience

• A degree in Computer Science, Data Engineering, Information Systems or a related discipline.
• Five to eight years of experience in data engineering, ETL/ELT development or data platform roles.
• Strong hands-on experience developing and deploying production solutions using AWS services.
• Proficiency in Apache Iceberg or similar open table formats.
• Experience in a Government Commercial Cloud (GCC) environment is strongly preferred.
• Relevant AWS certifications are an advantage.

Skills and Competencies

• Strong understanding of data lakehouse architecture, Apache Iceberg features and Amazon S3 Tables optimisation.
• Ability to design reliable pipelines and implement automated data quality and governance controls.
• Proficiency in production-quality coding, performance optimisation and cloud deployment.
• Ability to troubleshoot complex engineering issues and document solutions clearly.

/Ref: T24-251