Senior Site Reliability Engineer - Fleet
Lambda — San Francisco, California, United States · Remote · Senior
Role description
Role Overview
As a Senior Site Reliability Engineer, you will be responsible for building and operating monitoring and alerting for cluster health, remotely deploying and configuring large-scale HPC clusters, and troubleshooting and resolving cluster issues. You will work closely with on-site deployment teams and participate in on-call rotations. You will also contribute to and maintain Standard Operating Procedures and feed clear requirements back to other engineering teams.
What You Will Do
Your main day-to-day responsibilities will include building and operating monitoring and alerting for cluster health, remotely deploying and configuring large-scale HPC clusters, automating cluster lifecycle, creating runbooks and automated remediations for common cluster failure modes, and troubleshooting and resolving cluster issues.
Why It Might Be a Fit
To be a fit for this role, you will need to have 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role, a strong understanding of modern AI infrastructure, and experience with monitoring and alerting tools, automation and configuration management tools, and Linux-based systems in a distributed environment.
Requirements
- 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role
- Strong understanding of modern AI infrastructure
- Experience with monitoring and alerting tools (e.g., Prometheus, Grafana, Clickhouse)
- Proficiency in automation and configuration management tools (e.g., Ansible, Terraform)
- Experience with Linux-based systems in a distributed environment
- Solid understanding of Python and Go
- Excellent problem-solving and troubleshooting skills and an innate attention to detail
Benefits
- Generous cash & equity compensation
- Health, dental, and vision coverage for you and your dependents
- Wellness and commuter stipends for select roles
- 401k Plan with 2% company match (USA employees)
- Flexible paid time off plan