Software Engineer, Web Crawling

Exa(1 month ago)

SingaporeOnsiteFull TimeSenior$150,000 - $300,000Asia Engineering

About this role

The Software Engineer, Web Crawling at Exa is responsible for building and scaling a distributed web crawler to index the entire internet for a new search engine. This role involves designing systems that optimize crawl efficiency and politeness across numerous domains, tackling dynamic content and anti-bot measures, and developing algorithms for intelligent crawl scheduling. Candidates should have extensive experience in high-performance languages such as Rust or C++ and a strong interest in improving knowledge accessibility.

View Original Listing

Required Skills

Web Crawling
System Optimization
Distributed Systems
C++
Rust
Crawl Politeness
Rate Limiting
Dynamic Content Handling
JavaScript Rendering
Anti-Bot Measures

+2 more

About Exa

exa.ai

Exa is a state-of-the-art AI search engine and web crawling platform that provides robust APIs for retrieving real-time and structured web data. With its deep research tools, SERP API, and customizable web crawling capabilities, Exa empowers developers and enterprises to integrate high-quality search functionality directly into their applications. The platform is designed to enhance AI workflows by delivering comprehensive, relevant information from across the web while ensuring data privacy and compliance. Trusted by industry leaders, Exa offers a scalable solution tailored for diverse applications and business needs.