Data Engineer - Databricks & Automation
Must-have:PythonDataCI/CDAI
Key Responsibilities
- Design, develop and maintain scalable ETL/ELT data pipelines using Databricks, Python, PySpark, SQL and Delta Lake.
- Integrate data from multiple sources, including databases, APIs and files, with streaming ingestion where required.
- Develop and maintain curated datasets and data models to support business reporting, analytics and AI applications.
- Implement data quality checks, data validation, reconciliation and incremental data processing.
- Monitor data pipelines, investigate failures, troubleshoot issues and ensure data accuracy and availability.
- Optimise data processing performance, pipeline execution and compute utilisation.
- Implement version control, automated testing, CI/CD and technical documentation.
- Identify and automate repetitive data engineering and support activities.
- Explore practical AI