Lead Databricks Engineer - 12 months contract
Duration: 1 year Working Hours: 8.30am - 6pm, Monday - Fridays. Standby and shift work may be required Location: Central Key Responsibilities Lead the technical delivery of multiple Databricks initiatives, setting engineering standards, reviewing designs and code, and mentoring data engineers.
Act as the Databricks architect, translating business and technical requirements into secure, scalable and maintainable architectures on AWS.
Define Databricks Lakehouse architecture, workspace and environment strategy, data integration patterns, governance and security controls, compute strategy, and operational readiness.
Design and build production data pipelines using Python, PySpark, SQL and Delta Lake, covering batch, streaming and near-real-time processing using Medallion architecture across Bronze, Silver and Gold layers.
Lead hands-on development of data models, reusable engineering components and data-quality controls, including reconciliation, schema handling, recovery and reprocessing.
Design and enable production AI/ML and generative AI solutions on Databricks, including data preparation, feature and data pipelines, model integration, evaluation, deployment and monitoring, working with AI/ML specialists where appropriate.
Integrate Databricks with AWS services and enterprise systems, and work with platform and security teams on IAM, networking, storage and deployment architecture.
Optimise Spark workloads, compute and storage for performance, reliability and cost; resolve complex platform, pipeline and production issues.
Establish Git-based CI/CD, automated testing, observability and release practices for Databricks workloads, and maintain architecture and operational documentation.
Work with business stakeholders, architects and delivery teams to estimate effort, manage technical dependencies and communicate design decisions across initiatives.
Qualifications Bachelor's degree in Computer Science, Information Technology, Software Engineering or a related discipline.
10+ years of experience in data engineering, including enterprise data-platform design and delivery. 5+ years of substantial hands-on Databricks experience, including production implementation, optimization and troubleshooting.
2+ years serving as a Databricks architect or performing equivalent architecture responsibilities, including solution design and technical decision-making.
2+ year Hands-on experience delivering AI/ML or generative AI workloads on Databricks, with production implementation experience strongly preferred.
5+ years of hands-on AWS experience supporting data platforms and enterprise solutions.
Current Databricks certification is required, with Databricks Certified Data Engineer Professional strongly preferred. Additional Databricks certifications in machine learning or generative AI are advantageous.
Deep practical expertise in Apache Spark, Python/PySpark, SQL, Delta Lake, data modelling and lakehouse architecture, supported by strong data-engineering fundamentals.
Strong experience with Databricks orchestration, Unity Catalog, role-based access controls, metadata and lineage, data governance, performance tuning and production operations.
Practical experience supporting AI workloads on Databricks, including capabilities such as MLflow, Model Serving, model evaluation, vector search and retrieval-augmented generation, as relevant to prior projects.
Solid AWS expertise across S3, IAM, networking, security, compute and data integration, with the ability to select and integrate appropriate AWS services.
Experience with Git-based CI/CD, automated testing and deployment. Hands-on experience with Databricks Asset Bundles or equivalent Databricks deployment automation is strongly preferred; Terraform experience is advantageous.
AWS certifications are preferred, particularly in architecture or data engineering.
Able to lead multiple technical workstreams while remaining hands-on. Strong problem-solving, stakeholder communication and mentoring skills, with clear accountability for architecture quality, engineering standards and delivery outcomes. Interested candidate please click "APPLY" to begin your job search journey. By sending us your personal data, you are deemed to consent to PERSOL Tech Services Pte Ltd ("PERSOL") and its local and overseas subsidiaries and affiliates to collect, use and disclose your personal data for the purposes set out in the Privacy Policy. We may use artificial intelligence ("AI") and machine learning technologies to support our recruitment and placement services, including screening resumes, assessing qualifications and experience, and identifying skills and job matches. Where permitted by applicable law, we may also use such personal data to develop, train, test, improve and/or fine-tune AI and machine learning technologies that support our recruitment and business operations. For more information on how we collect, use, disclose and protect your personal data, please refer to our Privacy Policy: https://www.persoloutsourcing.com/policies . If you wish to withdraw your consent, please email us at Ken.Tan@persoloutsourcing.com PERSOL Tech Services Pte Ltd UEN No. 202233837K