Senior Network Production Engineer, AI Supercomputing

Overview

In this role, you will own the production health and operations of Microsoft’s MRC Ethernet backend network powering large-scale AI training. You will partner with cross-functional teams to ensure high availability, predictable performance, and rapid recovery from incidents, enabling researchers to run large jobs with minimal disruption. You’ll translate telemetry into actionable improvements, build automated mitigations, and scale the operating model for ever-larger training workloads. This is a hands-on, end-to-end role that blends network engineering, systems, and production operations to keep the AI training engine running smoothly. You’ll work in a fast-paced, on-call environment with a

Responsibilities

• Own production network health and SLIs for the MRC Ethernet backend
• Lead on-call responses and high-severity incidents
• Diagnose failures across GPUs, NICs, switches, firmware, drivers, and training workloads
• Develop automated mitigation, workload relocation, and quarantine mechanisms
• Create operational procedures for upgrades, maintenance, and topology changes
• Automate fleet operations, validation, monitoring, and change management
• Drive cross-team investigations and root-cause analyses with corrective actions
• Create runbooks and train engineers for 24/7 coverage
• Improve placement, routing, and reliability for large training jobs
• Establish end-to-end visibility from physical links to job health

Key requirements

• Significant experience operating large-scale datacenter or HPC networks in production
• Deep understanding of Ethernet, RDMA, RoCEv2, congestion control, and lossless network design
• Experience debugging complex failures across switches, NICs, hosts, drivers, firmware, and distributed apps
• Strong Linux and programming skills (Python, Go, C++, or similar)
• Experience building monitoring, automation, diagnostic, or remediation systems
• Proven ability to lead high-severity incidents and coordinate cross-team resolution
• Strong analytical skills and telemetry-to-root-cause translation
• Willingness to participate in a global 24/7 on-call rotation

• Clear communication under pressure
• Ownership and accountability
• Cross-functional collaboration
• Ethernet, RDMA, RoCEv2
• NCCL, CUDA, GPUDirect RDMA, distributed training frameworks
• Kubernetes, Slurm, workload placement

…