Senior Software Engineer (Agentic Search) - Crawler
Nebius — Amsterdam, North Holland, Netherlands · Hybrid · Senior
Role description
Role Overview
We are looking for a Senior Software Engineer to work on the content acquisition and crawling infrastructure of a novel search engine tailored for agentic AI consumption. You will design distributed crawling, scheduling, and ingestion infrastructure capable of operating at internet scale while balancing coverage, freshness, resource efficiency, and reliability.
What You Will Do
Design, implement, and operate web-scale crawling systems for acquiring content from the internet, build ingestion workflows for internal and external data sources, and develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
Why It Might Be a Fit
You may be a good fit if you have 5+ years of experience building backend or distributed systems, strong Go or C++ expertise, and experience with large-scale distributed systems.
Requirements
- 5+ years of experience building backend or distributed systems
- Strong Go or C++ expertise
- Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
- Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
- Experience operating production systems and debugging failures in distributed environments
- Strong understanding of scalability, fault tolerance, and resource management
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams