data engineer

Your tasks

JD

• Develop and maintain scalable data pipelines using PySpark and Spark SQL for processing large datasets efficiently.
• Write clean, reusable, and optimized code in Python for data manipulation, analysis, and automation tasks.
• Design and implement ETL workflows to extract, transform, and load data from various structured and unstructured sources.
• Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions.
• Optimize Spark jobs for performance tuning, resource utilization, and minimizing execution time.
• Work with distributed computing frameworks to process and analyze big data in a cloud or on-premises environment.
• Develop and maintain unit tests to ensure the accuracy and reliability of data pipelines and transformations.
• Utilize Spark SQL for querying and managing large datasets stored in distributed systems like Hadoop or cloud storage.
• Monitor and troubleshoot data pipeline issues, ensuring reliability and timely delivery of data.
• Stay updated with the latest advancements in PySpark, Spark SQL, and big data technologies to improve existing systems.

Requirements

• Big Data: 6 years (required)
• SQL: 5 years (required)
• PySpark: 1 year (required)

What we offer

Pay: $100,000.00-$130,000.00 per year

• Flexible schedule

Other

Work Location: Hybrid remote in Mississauga, ON (L5B)