Sr Data Engineer - Platform Modernization & Migration

About the company

How Jobgether works We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! Why Apply Through Jobgether?

Your tasks

• Data Pipeline Migration: Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, following established migration blueprints and ensuring reliable, consistent data delivery.
• Workflow Modernization: Upgrade orchestration workflows from Apache Airflow 2 to Airflow 3, including adapting, testing, and debugging DAGs to maintain scheduling reliability and operational continuity.
• Data Lake Modernization: Migrate legacy Hive-based data structures and datasets to Apache Iceberg, leveraging Snowpark Connect frameworks and connectors where applicable.
• Streaming Infrastructure Migration: Reconfigure and migrate Confluent and Apache Kafka streaming connections across cloud regions, ensuring seamless connectivity and minimal disruption to data flows.
• Pipeline Adaptation and Optimization: Adapt existing data pipelines for Snowflake environments, optimize SQL queries, and improve processing efficiency using advanced techniques such as common table expressions (CTEs), window functions, and complex joins.
• Data Processing and Orchestration: Build, maintain, test, and troubleshoot data workflows using Python, Apache Spark, PySpark, Apache Airflow, and Apache Kafka.
• Testing and Quality Assurance: Develop and execute testing strategies, perform hands-on debugging, and validate data accuracy, consistency, and integrity throughout the migration lifecycle.
• Performance and Reliability: Identify bottlenecks and optimize distributed data pipelines, storage structures, and query execution to meet performance and reliability standards.
• Cloud Data Management: Support data lake operations on AWS S3, including storage organization, bucket partitioning, and lifecycle management.
• Cross-Functional Collaboration: Partner with data engineers and platform teams to execute migration plans, resolve technical challenges, and deliver modernization milestones within established timelines.
• Independent Delivery: Manage assigned migration activities with a high degree of autonomy, communicating progress, risks, and technical findings clearly to distributed stakeholders.

Requirements

• Professional Experience: At least two years of hands-on data engineering experience working with large-scale datasets and distributed data processing environments.
• Programming and SQL: Strong Python development skills and advanced SQL expertise, including complex joins, CTEs, window functions, query optimization, and performance tuning.
• Data Warehousing: Proven experience with Snowflake, including data modeling, schema design, and adapting data pipelines to cloud data warehouse environments.
• Data Processing and Orchestration: Practical experience developing and testing pipelines with Apache Airflow and Apache Spark or PySpark, along with familiarity with Apache Kafka and streaming data architectures.
• Data Formats and Storage: Experience with Apache Iceberg and Apache Parquet, as well as AWS S3 and data lake concepts, including partitioning strategies and storage lifecycle policies.
• Testing and Debugging: Ability to design testing approaches, troubleshoot complex distributed systems, and independently resolve data pipeline issues.
• Cloud and Platform Modernization: Understanding of data platform migrations, workflow orchestration, and infrastructure modernization. Experience with Snowflake migrations or Apache Airflow version upgrades is a strong advantage.
• AI-Assisted Development: Familiarity with AI-powered coding and development tools to improve engineering productivity and accelerate delivery is desirable.
• Communication and Collaboration: Strong written and verbal English skills, with the ability to participate in technical discussions, produce documentation, and collaborate effectively with international teams.
• Autonomy and Adaptability: Demonstrated ability to work independently, prioritize tasks, and deliver results in a fast-paced, technically demanding environment.
• Remote Collaboration: Ability to work remotely while maintaining availability to overlap with the core engineering team until at least 12:00 PM Pacific Time. Candidates in EMEA or APAC are preferred, while the role is open to eligible candidates in Brazil.
• AWS WorkSpace Connectivity: Ability to access a client-provisioned AWS WorkSpace with reliable connectivity. Candidates must complete the connectivity test at https://clients.amazonworkspaces.com/Health and submit a screenshot of the results as supporting evidence with their application.
• Availability and Contract: Availability for a full-time, permanent position and willingness to collaborate across time zones as required.

What we offer

• Fully Remote Work: Work remotely from Brazil, with flexibility to operate from your preferred location, subject to the role's collaboration and time-zone requirements.
• Full-Time, Permanent Position: Opportunity to contribute to a long-term data platform modernization initiative.
• International Collaboration: Work alongside a globally distributed engineering team on a technically challenging project.
• Modern Data Technology Stack: Gain hands-on exposure to Snowflake, Apache Iceberg, Apache Airflow 3, Apache Spark, Kafka, and AWS cloud infrastructure.
• Large-Scale Engineering Impact: Contribute to the migration and modernization of approximately 1,000 datasets, helping improve data platform scalability, reliability, and performance.
• Technical Development: Strengthen expertise in cloud data warehousing, distributed processing, streaming infrastructure, data migration, and orchestration modernization.
• AI-Enabled Engineering: Opportunity to leverage AI-assisted development tools to improve productivity and streamline engineering workflows.

Other

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.