Staff engineer building web-scale crawling infrastructure for AI-native search.
Design and operate distributed systems that discover, fetch, and refresh content from the open web at internet scale. You'll build crawl scheduling, URL deduplication, and ingestion pipelines handling billions of URLs while optimizing for freshness, coverage, and efficiency. This role requires deep expertise in distributed systems, Go or C++, and production infrastructure; it suits senior engineers with experience in large-scale data platforms or search technology.
Membership is €29/month, cancel anytime: every rate, every original listing link, and a daily alert for roles matching your filters.
Found at a specialist agency · listed 17 October 2026 · InsideJobs links you to the original posting.