Site Reliability Engineer

About The Role

We're seeking an experienced Network Engineer to design, build, and optimize the high-performance networking infrastructure powering our AI/ML operations in Toronto. You'll work at the cutting edge of network technology—managing InfiniBand and ultra-high-speed Ethernet fabrics that connect NVIDIA H100 and A100 GPUs, over 20PB of Ceph storage, and hundreds of servers.

You'll be hands-on with the full lifecycle of our network infrastructure: planning, building, testing, deploying, and keeping everything running at peak performance. That means troubleshooting issues as they arise, monitoring network performance and throughput, developing automation to streamline operations, and working closely with HPC and ML teams to ensure they have the bandwidth they need. You'll also help us plan for future capacity and evaluate emerging network technologies as we scale to meet increasingly demanding workloads.

Responsibilities Design, operate, and improve reliable infrastructure for AI training and inference workloads

Own and automate operational workflows across one or more core areas: networking, compute allocation, storage, GPU/server configuration, or AI platforms

Build monitoring, alerting, runbooks, and incident-response practices that make systems easier to operate

Diagnose performance, capacity, and reliability issues across hardware, operating systems, networks, schedulers, and distributed workloads

Partner closely with ML, research, and platform teams to translate workload needs into practical infrastructure improvements

Improve provisioning, configuration management, testing, and deployment automation

Help plan cluster growth, capacity allocation, upgrades, and lifecycle management

Contribute to a thoughtful reliability culture through documentation, post-incident learning, and pragmatic engineering standards

Minimum Qualifications 4+ years of experience in site reliability engineering, infrastructure engineering, systems engineering, or a related production-operations role

Strong hands-on expertise in at least one of the following:

Networking, including firewalls, switching, routing, ASN/BGP configuration, or InfiniBand

Cluster and systems allocation with Kubernetes, SLURM, MAAS, or similar platforms

Distributed storage, particularly Ceph

GPU and server administration, including CUDA drivers, firmware, BIOS, and hardware troubleshooting

AI training or model-serving infrastructure

Experience operating production systems with a focus on availability, performance, security, and automation

Strong Linux administration and scripting skills

A systematic approach to troubleshooting across multiple layers of a complex system

Clear written and verbal communication skills, including the ability to work effectively with a distributed team

Preferred Qualifications Experience supporting GPU-intensive AI or HPC environments

Experience with NVIDIA GPUs, CUDA, NCCL, and high-performance interconnects - Experience with InfiniBand, RDMA, RoCE, or 100Gb+ Ethernet

Familiarity with Kubernetes, SLURM, MAAS, Terraform, Ansible, or similar infrastructure tooling

Experience operating or tuning Ceph clusters

Familiarity with observability tooling such as Prometheus, Grafana, and centralized logging systems

Experience with hardware provisioning, firmware management, and bare-metal automation

Experience running large-scale distributed training or high-throughput inference workloads

Familiarity with cloud and hybrid infrastructure across AWS, GCP, or Azure

If you're a natural problem-solver with a passion for continuous learning, we'd love to hear from you.