Senior Data Engineer (ETL/ELT - Python, PySpark & SQL)
Must-have:PythonCloudDevOpsDataAISecurityJunior
Responsibilities
- Design, develop and maintain data pipelines for data ingestion, transformation and consumption.
- Build and optimise ETL/ELT processes using Python,PySpark and SQL.
- Develop and support data solutions on Snowflake, Data bricks and Cloudera Data Platform.
- Perform data migration from legacy systems to modern data platforms.
- Develop data integration solutions across enterprise applications and databases.
- Perform data cleansing, transformation, validation and quality checks.
- Optimise data pipelines for performance, scalability and reliability.
- Support AI/ML integration and data preparation requirements.
- Implement data security, encryption and masking requirements.
- Support automated testing, deployment and production troubleshooting.
- Prepare technical documentation and mentor junior data engineers.
Requirements
- Bachelor's degree in Computer Science, IT, Engineering or related discipline.
- Minimum 5 - 8 years of experience in data engineering, ETL development or big data technologies.
- Strong programming skills in Python, PySpark andSQL.
- Hands-on experience with Snowflake, Data bricks and/or Cloudera Data Platform.
- Good understanding of data lakes, data warehouses and ETL/ELT processes.
- Experience with relational databases such as Oracle, SQL Server, MySQL or PostgreSQL.
- Knowledge of Informatica Power Center or other data integration tools.
- Familiarity with Autosys, Airflow or other scheduling tools.
- Knowledge of cloud platforms and DevOps practices.
- Understanding of data governance, data security and data quality.
- Good analytical, troubleshooting and communication skills.