Sr. Data Engineer - Spark

Building high-scale batch and near-real-time data pipelines deployed on infrastructure we run ourselves (on-prem), not managed cloud services. You will design and operate high-volume analytical data systems end to end, with Apache Spark as the core processing engine for both batch and streaming workloads.

7+ years of experience in data engineering and software development

Ability to write high-quality code in Java/Scala, Python, or equivalent languages

Deep, hands-on production experience with Apache Spark batch and Spark Structured Streaming (core requirement)

Demonstrated Spark performance tuning: partitioning, caching and persistence, broadcast joins, shuffle reduction, data-skew handling, and Adaptive Query Execution

Experience operating Spark on self-managed clusters (YARN, Kubernetes, or standalone) — executor sizing, resource allocation, and multi-tenant workloads

Practical experience with Kafka (or equivalent messaging systems) as a Spark source and sink for high-volume workloads, including offset and checkpoint management

Practical experience with distributed query engines (e.g., Trino/Presto or similar)

Practical experience with ETL / data integration tools, commercial or open-source (e.g., Datastage, Informatica, Apache NiFi, or similar)

Practical experience with SQL-based transformation frameworks (e.g., dbt or others)

Strong SQL skills and understanding of data modeling and data warehousing for analytical workloads

Hands-on experience with real-time / low-latency analytical stores (columnar or OLAP engines, e.g., Apache Pinot/ClickHouse or similar)

Practical experience with big-data platforms and distributions (e.g., Cloudera, Hadoop ecosystem, Databricks, or similar)

Practical experience containerizing and operating data workloads (Docker; Kubernetes a plus)

Experience with workflow orchestration tools (e.g., Airflow or similar)

Familiarity with data lake table formats (e.g., Apache Iceberg, Delta Lake, or similar), including schema evolution and compaction

Familiarity with data governance / cataloging tools (e.g., DataHub or similar)

Familiarity with lakehouse management systems (e.g., Apache Amoro or similar)

Familiarity using AI tools for development and debugging (Claude, Cursor, Codex)

Key Skills Apache Spark, Spark SQL, Spark Structured Streaming, PySpark, Airflow, SQL, ETL, Java, Scala, Python, Apache Iceberg, Delta Lake, Data Lakehouse, Trino, Data Lake, Data Warehousing, Data Modeling, Data Architecture, Kafka, Hadoop, Cloudera, Databricks, Docker, Kubernetes