Lead Data Engineer (Teradata/Hadoop)

Company Overview

OPTIMUM SOLUTIONS (SINGAPORE) PTE LTD is a Singapore-headquartered IT outsourcing firm specializing in software engineering, application development, and IT infrastructure management. Since 1997, it has grown into a leading provider of digital transformation services.

Job Summary

We are seeking a Senior Data Engineer with expertise in enterprise data platforms and banking domains to lead the design and development of scalable data pipelines across Teradata, Hadoop/Cloudera, and cloud ecosystems while mentoring engineering talent and driving technical excellence.

Responsibilities

• Lead the architecture and implementation of scalable batch, streaming, and API-based data pipelines to support large-scale data delivery and historical data processing frameworks.
• Design and implement Slowly Changing Dimension (SCD) and Change Data Capture (CDC) mechanisms to maintain accurate and up-to-date data products.
• Develop semantic-layer data products that enhance data accessibility and usability for business stakeholders.
• Optimise complex SQL queries, Spark workloads, partition strategies, and storage frameworks to improve performance for large-volume data processing in regulated enterprise banking environments.
• Implement infrastructure-as-code using Terraform to automate provisioning and management of cloud and on-premises resources.
• Establish automated Continuous Integration/Continuous Deployment (CI/CD) pipelines and containerized deployment workflows to ensure reliable and repeatable software delivery.
• Enforce rigorous testing standards to maintain high-quality, production-ready data engineering solutions.
• Lead code reviews and enforce engineering best practices to uphold technical excellence across teams.
• Mentor junior engineers to develop their technical skills and promote disciplined delivery in regulated settings.
• Collaborate closely with architects, product teams, DevOps, and business stakeholders to align data engineering efforts with organizational goals.

Required competencies and certifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related discipline (required education).
• At least 10 years of data engineering experience with technical leadership in Teradata and Hadoop/big-data delivery.

Preferred competencies and qualifications

• Certifications in Teradata, Cloudera, Databricks, Cloud Data Engineering, or Terraform.
• Proficiency with core technologies including Teradata, Hadoop/Cloudera, Spark, Hive, Impala, HBase, SQL, Python, Scala/Java, Kafka, Airflow, and Denodo.
• Experience with infrastructure and DevOps tools such as Terraform, Kubernetes/OpenShift, Git, and CI/CD automation tools.
• Domain expertise in enterprise banking data including Customer, Payments, Risk, Finance, AML, and Regulatory Reporting, with a focus on data quality, reconciliation, lineage, and historical reconstruction.
• Demonstrated technical leadership, disciplined delivery in regulated environments, strong communication skills, and accountability for production-ready outcomes.