Senior Data Engineer (Permanent)
Must-have:JavaGitDockerKubernetesDataCI/CDSenior
About the Role
We are seeking an experienced Senior Data Engineer to design, develop and maintain large-scale data processing solutions for aviation systems. You will build reliable batch and real-time data pipelines, integrate multiple data sources and implement scalable data storage, processing and recovery mechanisms.
Key Responsibilities
- Design, develop and maintain batch and real-time data pipelines using Apache Spark, Apache Flink and Apache Kafka.
- Implement distributed data processing solutions using Java or Scala and SQL.
- Integrate multiple data sources and S3-compatible object storage for data ingestion and processing.
- Manage data schemas, metadata, lineage and data quality across processing pipelines.
- Deploy, maintain and scale data processing workloads in Kubernetes environments.
- Implement automated testing, GitLab CI/CD pipelines, monitoring and operational troubleshooting.
- Establish checkpointing, recovery, data retention and lifecycle management mechanisms.
- Support high availability, backup, disaster recovery and data synchronisation across multiple sites.
Requirements and Qualifications
- 5–8 years of relevant experience in data engineering, distributed systems or large-scale data platform development.
- Strong hands-on experience designing and developing batch and streaming data pipelines using Apache Spark, Apache Flink and Apache Kafka.
- Proficiency in Java or Scala for distributed data processing, together with strong SQL skills.
- Experience deploying and scaling data workloads using Docker and Kubernetes.
- Familiarity with S3-compatible object storage, data schemas, metadata management, data lineage and data quality controls.
- Experience with checkpointing, fault recovery, data retention and disaster recovery mechanisms.
- Hands-on experience with GitLab, CI/CD pipelines, automated testing and monitoring.
- Strong analytical, troubleshooting and problem-solving skills.
Only shortlisted candidates will be notified.
(EA Reg No: 20C0312)