Databricks Engineer – 12 months contract

Key Responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

Requirements

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and layered data architecture such as Bronze/Silver/Gold.
• Experience building production data pipelines with proper logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis, or related services.
• Good understanding of data modelling, data quality, and performance optimisation.
• Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Databricks, AWS, data engineering, AI/ML, or GenAI-related certification(s).
• Strong engineering fundamentals and willingness to learn.
• Practical problem-solving and attention to data correctness.
• Clean implementation, documentation discipline, and reliable delivery.
• Ability to work in a fast-paced project environment and collaborate with different technical teams.
• Ownership of assigned tasks and openness to technical guidance and feedback.