Senior Software Engineer (Agentic Search) - Crawler
Nebius — Zurich, Zurich, Switzerland · In-Person · Senior · $100k - $200k/yearly
Role description
Role Overview
We are building an agent-native search platform designed specifically for AI systems rather than human users. Our product provides programmatic, low-latency, and observable search APIs that AI agents use to retrieve, filter, and reason over real-world information at scale.
What You Will Do
Design, implement, and operate web-scale crawling systems for acquiring content from the internet, build ingestion workflows for internal and external data sources, and develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
Why It Might Be a Fit
You will be a good fit if you have 5+ years of experience building backend or distributed systems, strong Go or C++ expertise, and experience with large-scale distributed systems, web protocols, and content extraction.
Requirements
- 5+ years of experience building backend or distributed systems
- Strong Go or C++ expertise
- Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
- Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
- Experience operating production systems and debugging failures in distributed environments
- Strong understanding of scalability, fault tolerance, and resource management
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams