Senior Observability Platform Engineer

Role description

Role Overview

As a Senior Observability Platform Engineer, you'll play a key role in designing, building, and scaling Nscale's observability platform. You'll focus on delivering reliable, high-quality visibility into GPU clusters, AI workloads, and the infrastructure that powers them. You'll approach observability as a product—balancing usability, scalability, and operational efficiency. You'll build systems that reduce cognitive load for engineers, surface meaningful signals, and enable fast, confident debugging when things go wrong.

What You Will Do

Design, build, and operate scalable observability systems across metrics, logs, traces, and alerting. Contribute to architectural decisions around tooling, data pipelines, storage, and retention strategies. Improve signal quality by reducing noise, managing cardinality, and refining alerting practices. Help identify and address observability gaps before they impact reliability.

Why It Might Be a Fit

You'll contribute to platform direction, implement critical systems, and collaborate closely with SRE, infrastructure, and AI/ML teams to ensure observability is embedded into everything we run. This is a hands-on engineering role with meaningful influence over platform design and evolution.

Requirements

• 5+ years in SRE, infrastructure engineering, platform engineering, or observability-focused roles
• Experience operating and scaling observability systems in production environments
• Strong understanding of monitoring concepts: metrics, logs, traces, alerting, and SLOs
• Hands-on experience with several of: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic
• Solid programming skills (Python, Go, or similar) with the ability to build and maintain production systems
• Experience working with Kubernetes-based infrastructure
• Familiarity with Infrastructure-as-Code (Terraform, Ansible, or similar)
• Pragmatic mindset with a focus on simplicity, reliability, and maintainability
• Strong collaboration skills and ability to work across teams

Benefits

• Medical
• Dental
• Vision
• Flexible paid time off
• Parental leave
• Retirement plan participation