Devops Engineer - Databricks

Job Description

• We are looking for an experienced Databricks Infrastructure & DevOps Engineer to manage, optimize, and secure enterprise Databricks environments hosted on AWS.
• The ideal candidate will have strong expertise in cloud infrastructure, DevOps automation, CI/CD pipelines, Infrastructure as Code (IaC), and security governance within highly regulated environments.

Key Responsibilities

Infrastructure & Platform Management

• Design, deploy, and administer Databricks workspaces and clusters on AWS.
• Build and manage cloud infrastructure using Terraform and Databricks Asset Bundles (DAB).
• Configure and optimize AWS services including S3, IAM, VPC, Security Groups, and networking.
• Monitor platform performance, availability, scalability, and reliability. Manage secrets, encryption keys, and platform security configurations.
• Implement backup, retention, and disaster recovery strategies.

DevOps & CI/CD

• Design, develop, and maintain CI/CD pipelines for Databricks deployments.
• Manage source control and automated deployments using GitLab.
• Implement code quality and security scanning using SonarQube, Fortify on Demand (FOD), Nexus IQ, and CloudSCAPE.
• Automate infrastructure provisioning and application deployment.
• Support release management and deployment automation across environments.

Security & Compliance

• Ensure compliance with government security frameworks and data governance standards.
• Support vulnerability assessments, penetration testing (VAPT), and remediation activities.
• Manage identity and access controls through Central Accounts Management (CAM). Implement security monitoring, vulnerability management, audit logging, and compliance reporting.
• Coordinate security testing, System Security Acceptance Testing (SSAT), and regulatory compliance activities.

Collaboration & Support

• Collaborate with Data Engineers, Data Analysts, Cloud Architects, and Security teams to support enterprise data platforms.
• Support multiple Databricks environments and data pipelines.
• Provide technical guidance on infrastructure, deployment automation, and platform optimization.
• Maintain technical documentation, operational procedures, and knowledge-sharing materials.
• Support environment provisioning, onboarding, and training activities.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Minimum 3 years of experience in DevOps, Cloud Infrastructure, or Platform Engineering.
• Hands-on experience administering the Databricks platform.
• Strong experience with AWS Cloud services.
• Proficiency in Terraform and Infrastructure as Code (IaC).
• Experience designing and managing GitLab CI/CD pipelines.
• Strong scripting skills using Python, Bash, or PowerShell.
• Experience implementing security best practices in enterprise cloud environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Technical Skills

• Databricks Administration
• AWS Cloud (S3, IAM, VPC, Security Groups)
• Terraform
• Databricks Asset Bundles (DAB)
• GitLab CI/CD
• SonarQube
• Fortify on Demand (FOD)
• Nexus IQ
• CloudSCAPE
• Python / Bash / PowerShell
• Infrastructure as Code (IaC)
• Security Monitoring & Compliance Vulnerability Management
• Containerization & Orchestration

Preferred Qualifications

• Experience working in Government Cloud (GCC) or highly regulated environments.
• Knowledge of government security standards, data classification, and compliance frameworks.
• Experience supporting enterprise data platforms and Databricks environments.
• Familiarity with SSAT, VAPT, PDPIA, and security governance processes.

Preferred Certifications

• Databricks Certified Associate (or higher)
• AWS Certified DevOps Engineer
• AWS Certified Solutions Architect (Preferred)