Principal Infrastructure Engineer, AI Cluster Performance & Validation

Role description

Role Overview

As a Principal Infrastructure Engineer, AI Cluster Performance & Validation, you will be a critical member of the AI Infrastructure Operations team, responsible for ensuring the acceptance, performance, and scalability of our cutting-edge AI and High-Performance Computing (HPC) environments.

What You Will Do

Own the technical definition of 'healthy at scale.' Set the architecture, roadmap, and acceptance criteria by which multi-thousand-GPU clusters are declared production-ready, and establish the performance bar (collective bandwidth, job goodput, model FLOPs utilization) that every cluster must clear before and after customer handover.

Why It Might Be a Fit

This role requires a deep understanding of high-performance fabrics, Linux systems expertise, and strong production Python, plus experience with C/C++ or Go and with configuration management tooling.

Requirements

• Bachelor's or higher degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.
• 10+ years of relevant experience building, operating, or debugging large-scale compute infrastructure, including significant time at staff or principal level owning cross-team technical direction.
• AI Workload Expertise: Hands-on experience running real AI compute jobs at scale — pre-training, fine-tuning, or large-scale inference of open-source or proprietary models — including practical familiarity with distributed training strategies and frameworks such as PyTorch, Megatron-LM, DeepSpeed, or equivalent.
• Cluster Validation: Demonstrated experience validating and accepting large clusters (thousands of GPUs) for performance and reliability, with a working command of benchmark methodology and the ability to defend a number to both engineers and customers.
• Performance Debugging: Proven ability to diagnose distributed performance problems — stragglers, collective stalls, link flaps, thermal throttling, silent data corruption, ECC and Xid errors, noisy-neighbor and storage-bound bottlenecks — using tools such as NCCL debug tracing, Nsight Systems/Compute, PyTorch Profiler, perf, and fabric telemetry.
• Networking: Deep understanding of high-performance fabrics — InfiniBand and/or RoCEv2, RDMA, GPUDirect, adaptive routing, congestion control, and rail-optimized topologies — and of networking fundamentals (TCP/IP, BGP).
• Systems & Programming: Deep Linux systems expertise (kernel tunables, NUMA, PCIe, IRQ and memory behavior) and strong production Python, plus experience with C/C++ or Go and with configuration management tooling (e.g., Ansible, Terraform).
• Schedulers: Experience operating and debugging AI workloads under SLURM and/or Kubernetes at scale.