Senior Data Engineer - IT Contracting

Our client is a global pharmaceutical company that develops advanced data & analytics solutions supporting business and regulatory processes on an international scale. We are looking for an expert to join the team responsible for building and developing a strategic data platform in an Azure Databricks environment.

Industry: Technology & Telecoms Area: IT Data Analysis We offer

Rate up to 34 EUR/h

Fully remote work

Project start: November 1st

Long-term project until 31.10.2027

Work in an international environment on the development of a strategic data platform

Real impact on the development of modern solutions based on Azure Databricks and Lakehouse architecture

B2B contract via Michael Page

Daily tasks

• Designing, developing, and maintaining efficient data processing processes in the Azure Databricks environment.
• Creating and developing data pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• Implementing data integration processes from API interfaces, JSON files, Azure Storage services, and corporate systems.
• Implementing and developing data architecture based on the Bronze, Silver, Gold model.
• Transforming, standardizing, and preparing data for further use by business and system consumers.
• Building data quality control mechanisms, validation, data reconciliation, and monitoring of processing processes.
• Creating automated data tests and implementing operational quality control mechanisms.
• Optimizing environment performance, processing processes, and data partitioning strategies.
• Implementing CI/CD processes for code and data engineering solutions.
• Creating monitoring, error handling, and data reprocessing mechanisms.
• Maintaining high standards of data management, compliance, traceability, and auditability of processes.
• Collaborating with architects, business stakeholders, and technology teams in designing and implementing new solutions

Requirements

Several years of experience in a Data Engineer position or a similar role.

Very good knowledge of Azure Databricks.

Practical experience working with Python, PySpark, and Spark SQL.

Knowledge of Delta Lake and Lakehouse architecture.

Experience in designing and developing solutions based on the Bronze, Silver, Gold model.

Ability to build data integration processes using APIs, files, and Azure cloud services.

Experience working with ADLS Gen2 and data storage services in the Azure ecosystem.

Knowledge of issues related to data quality, schema validation, data reconciliation, and process monitoring.

Experience in metadata management and data lineage tracking.

Practical knowledge of Unity Catalog or similar solutions in the governance area.

Ability to optimize the performance of processes and large data volumes.

Experience in implementing CI/CD and automated tests for data engineering solutions.

Experience in environments with high requirements for compliance, quality, and data control.

Very good knowledge of the English language.

Nice to have:

Experience with Databricks Workflows and Databricks Jobs.

Knowledge of Delta Live Tables or Lakeflow.

Experience related to processing document metadata and managing references to electronic resources.

Experience in projects requiring full traceability, auditability, and controlled deployment processes.

Must have: Azure Databricks, Python, PySpark, Spark SQL, API, Azure, ADLS, Data engineering

Nice to have: Databricks, Delta Live Table