Lead Data Engineer
Primary Skills
ETL Fundamentals, SQL, SQL (Basic + Advanced), Python, Data Warehousing, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures
Specialization
Databricks Engineering: Data Engineer
Job requirements
Job Description – Data Engineer (Databricks)
Company: Brillio Technologies
Role: Data Engineer – Databricks
Experience: 5–10 Years
Location: India (Hybrid/3 days)
Employment Type: Full-time
About Brillio
Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.
Role Overview
We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.
Key Responsibilities
Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)
Develop data ingestion frameworks from multiple sources (batch & real-time)
Implement Delta Lake features including schema evolution, ACID transactions, and optimization
Create and manage Databricks notebooks, workflows, and jobs for production pipelines
Ensure data quality, governance, and lineage across pipelines
Optimize Spark jobs for performance (partitioning, caching, tuning)
Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
Support production deployments, monitoring, and troubleshooting of data pipelines
Contribute to migration of legacy data platforms to modern cloud-based architecture
Technical Skills (Must-Have)
Strong experience in:
Databricks (Workflows, Notebooks, Jobs)
PySpark / Spark SQL
SQL & Data Warehousing concepts
Hands-on experience with:
Delta Lake / Lakehouse architecture
Data pipeline development & orchestration
Good understanding of:
Data modeling, ETL/ELT processes, SCD concepts
Cloud & Tools
Experience in at least one:
AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)
Workflow tools:
Airflow / Databricks Workflows
Version control:
Git / CI-CD pipelines
Good to Have
Experience with:
Streaming frameworks (Kafka / Spark Streaming)
DBT / Snowflake / Redshift / Fabric
Exposure to:
Unity Catalog, Autoloader, Delta Live Tables
Experience Required
5–10 years of experience in Data Engineering
Minimum 2–4 years of hands-on Databricks experience
Proven experience building enterprise-scale data platforms and pipelines
Soft Skills
Strong problem-solving and analytical skills
Ability to work in a fast-paced, agile environment
Good communication and stakeholder management
Why Join Brillio
Work on cutting-edge data & AI programs
Opportunity to build large-scale data platforms
Collaborative and innovation-driven culture