Staff Software Engineer, Inference

Role description

CoreWeave is hiring a Staff Software Engineer for its Inference team. The role involves leading cross-team design initiatives, optimizing inference performance, and improving system reliability at scale. The team builds and operates CoreWeave’s Kubernetes-native inference platform, powering low-latency, high-throughput AI workloads.

Requirements

• 8–12+ years of experience building and operating large-scale distributed systems or cloud platforms
• Proven experience leading cross-team technical initiatives impacting multiple services or organizations
• Strong programming skills in Go, Python, or C++
• Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design
• Strong understanding of distributed systems, networking, and performance optimization
• Experience designing and operating low-latency, high-throughput systems with strict P95/P99 latency requirements
• Hands-on experience with inference systems, including batching or micro-batching strategies, caching, and memory optimization
• Experience improving system performance using metrics-driven approaches (e.g., latency, throughput, utilization)
• Familiarity with mixed precision (BF16, FP8) and streaming inference workloads
• Experience with inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe
• Experience with GPU systems and performance optimization (CUDA, NCCL, RDMA, NUMA, GPU interconnects)
• Experience leading multi-team or org-level technical initiatives
• Exposure to large-scale AI/ML infrastructure or hyperscale cloud environments

Benefits

• Medical, dental, and vision insurance
• Company-paid Life Insurance
• Voluntary supplemental life insurance
• Short and long-term disability insurance
• Flexible Spending Account
• Health Savings Account
• Tuition Reimbursement
• Ability to Participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations