Databricks Data Engineer

About YellowIpe

Our mission is to inspire the connection of technology with people; we foster the best in our professionals through our expertise in finding and attracting the best talent for the best projects. Focus on People, Collaboration, and Commitment are the pillars that guide us on this journey.

Join the yellow team as our new Databricks Data Engineer!

Responsibilities:

Collaborate with the Data & Analytics team of a prestigious financial institution in Lisbon, supporting the migration of data engineering capabilities from Azure Synapse to the Databricks platform. Execute data engineering tasks, focusing on the effective adoption and implementation of Databricks, using the established Data Vault architecture that classifies data into bronze layers (raw data), silver layers (structured and processed data that compose the data lake layer), and gold layers (business-oriented and consumption-ready layer). Build, optimize, and maintain data transformation pipelines, ensuring efficiency and scalability in data movement between the various layers. Implement robust data governance practices during the engineering process, ensuring the creation of a comprehensive data catalog and data lineage, integrating these functionalities into the new architecture. Work in close collaboration with the internal team, providing technical support and sharing knowledge to strengthen the team's engineering capabilities. Perform rigorous testing and ensure data quality throughout the migration process, proactively addressing and resolving performance issues. Contribute to technical documentation and the best practices manual, ensuring that all knowledge acquired during the project is accessible to the team.

Requirements:

Proven hands-on experience with Databricks; generic experience in Spark or cloud data engineering is not sufficient (mandatory). Solid knowledge in Data Engineering, with a demonstrated track record in building and optimizing pipelines in layered data architectures (mandatory). Previous experience in data governance practices, including the implementation and management of data catalogs and data lineage (mandatory). Familiarity with Data Vault modeling will be considered a significant advantage. Between 4 to 5 years of relevant experience in Data Engineering, preferably in financial or similar environments. Fluency in English is essential; knowledge of Portuguese will be a valued differentiator. Availability for a hybrid work regime, with at least 2 days per week requiring physical presence in the office. Location: Lisbon, Portugal.

Differentials:

Experience with Databricks' Artificial Intelligence and Machine Learning capabilities will be considered an advantage, although it does not constitute an essential requirement. Ability to work in cross-functional teams and agile environments, adapting quickly to new technologies and challenges. Certifications in Databricks, Azure, or other relevant platforms are a plus.

Important information:

Contract type: full-time Location: Lisbon, Portugal Modality: Hybrid – at least 2 times per week in the office Required language(s): English mandatory; Portuguese will be considered a differentiator Level: Not specified

Application: View the vacancy and apply at YellowIpe