Senior Principal Site Reliability Engineer, Infrastructure & Platform

We are looking for a Senior Site Reliability Engineer that leads with kindness, and possesses a strong software development background to join our Infrastructure Engineering team. Your primary focus will be building automation, tooling, and internal platforms that enable our team to operate a global, multi-datacenter infrastructure spanning a growing number of Points of Presence across the globe. Deep familiarity with production infrastructure -- bare-metal hypervisors, containerized workloads, Kubernetes clusters, and cloud platforms -- is essential, but your primary lens should be on automation and code.  You will develop internal tools and APIs in Python and Go, design and maintain Ansible automation across hundreds of hosts, build CI/CD pipelines, and create self-service interfaces that reduce toil and eliminate manual operations. You will work within a PCI-DSS compliant environment and participate in a 24x7 on-call rotation. Responsibilities: Internal Tooling & Application Development  Design and develop internal tools, CLIs, and APIs (primarily in Go and Python) that enable infrastructure self-service, automate complex workflows, and improve operational efficiency 

Build integrations between infrastructure systems -- connecting CMDB/IPAM (NetBox), secrets management (HashiCorp Vault), hypervisor APIs (Proxmox), monitoring platforms, and CI/CD pipelines into cohesive automated workflows 

Develop and maintain API clients and libraries for interacting with infrastructure services (Proxmox API, Vault API, NetBox API, iLO Redfish, container registries) 

Write well-tested, documented, and maintainable code with proper versioning, release processes, and code review practices 

Infrastructure as Code & Ansible Development   Architect, develop, and refactor Ansible roles and playbooks across a large-scale inventory spanning 30+ datacenters, 80+ group variable files, and 40+ roles 

Design reusable, composable Ansible role patterns that scale cleanly as the DC footprint grows -- new DCs should be deployable with minimal variable additions 

Improve idempotency, error handling, and test coverage across the existing Ansible codebase 

Develop custom Ansible modules, plugins, and lookup plugins where upstream modules may be insufficient (e.g., custom Vault integration, Proxmox API interactions, iLO automation) 

Automate bare-metal server lifecycle end-to-end: from iLO bootstrap through OS installation, hypervisor configuration, VM provisioning, and service deployment 

CI/CD Pipeline Engineering   Design, write, and maintain GitLab CI pipelines for infrastructure automation, including multi-stage deployment workflows with linting, validation, canary testing, and regional rollout 

Build pipeline patterns for safe infrastructure changes: staged rollouts, automated rollback, drift detection, and change validation 

Create reusable pipeline templates and shared CI components that standardise how infrastructure changes are tested and deployed 

Implement automated testing for Ansible roles and infrastructure changes (molecule, ansible-lint, integration testing in ephemeral environments) 

Kubernetes & Container Platform Automation   Develop automation for self-hosted Kubernetes cluster lifecycle management: provisioning, upgrades, scaling, and disaster recovery 

Build and maintain container image build pipelines, registry management, and image promotion workflows 

Create Kubernetes operators or controllers (in Go) where custom automation of cluster-level concerns is needed 

Automate workload deployment patterns, including Helm chart development and GitOps workflows 

Cloud Infrastructure Automation   Develop IaC and automation for AWS and Azure resources, integrating cloud infrastructure with on-premises systems 

Build automation that spans hybrid environments -- coordinating deployments across bare-metal, virtualized, and cloud targets from a unified pipeline 

Observability & Reliability Engineering   Instrument internal tools and automation with proper logging, metrics, and tracing 

Build automated remediation workflows that respond to monitoring alerts and reduce mean time to recovery 

Develop reporting and dashboards that provide visibility into infrastructure state, automation success rates, and toil metrics 

Identify and automate away recurring operational toil; track and quantify toil reduction over time 

Security & Compliance Automation   Automate PCI-DSS compliance workflows including CIS benchmark hardening, audit evidence collection, and configuration drift detection 

Build automated secret rotation pipelines using HashiCorp Vault 

Develop security scanning integration into CI/CD pipelines (container image scanning, infrastructure configuration validation) 

What We're Looking For   5+ years of experience in an SRE, DevOps, or Infrastructure Engineering role with a strong emphasis on writing code and building automation 

Proficiency in Python, with experience building CLI tools, APIs (Flask/FastAPI or equivalent), and automation frameworks 

Expert-level Ansible skills: custom role development, module/plugin authorship, complex Jinja2 templating, inventory management at scale, and CI/CD integration 

Solid Linux systems knowledge (RHEL/CentOS) -- you need to understand the systems you're automating at a depth that lets you debug failures and design robust automation 

Experience building and maintaining CI/CD pipelines (GitLab CI preferred) for infrastructure automation, not just application builds 

Production experience with self-hosted Kubernetes: cluster operations, controller/operator development, and workload automation 

Practical AWS and Azure experience with an IaC mindset -- provisioning and managing cloud resources through automation, not console clicks 

Experience with API-driven infrastructure management (RESTful APIs, Redfish/iLO, hypervisor APIs) 

Familiarity with HashiCorp Vault or equivalent secrets management platforms, including programmatic integration 

Understanding of PCI-DSS requirements as they apply to automated infrastructure management -- audit trails, change control, hardening automation 

Strong software engineering fundamentals: version control workflows, code review, testing practices, documentation, and release management