Automation Testing Engineer

Experience Requirements

Total QA/Testing Experience: 5+ years.

Data Testing Experience: 3+ years specifically in Big Data, Hadoop, or CloudData Warehouse environments.

Good to have : Databricks Experience: 1+ years of experience testing pipelines within a Databricks environment.

Automation Focus : Proven track record of moving from manual SQL checks to automated Python-based testing frameworks.

Migration automation testexperience using Python

Required Certifications

Good to have: Databricks Certified Data Engineer Associate (at minimum).

Preferred: ISTQB Foundation or Advanced Level (Test Automation Engineer).

Core Technical Skills

1. Data Validation Frameworks

• Great Expectations / Pandera: Proficiency in using Python-based libraries to define data contracts and automated validation suites.
• DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records).
• Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records.

1. Python for QA (PySpark)

• Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic.
• Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold).
• Data Reconciliation: Building Python scripts to perform source-to-target counts and checksums across distributed file systems.

1. Performance Integration Testing

• Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes.
• End-to-End Orchestration Testing: Testing the reliability of Databricks Workflows and handling of job failures/retries.
• Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables.

1. Governance Security Testing

• Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data.
• Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse.

EANumber: 11C4879

Reg. ID : R26161692