Software Engineer, Web Crawling

Exa

3 months ago

3.4 on Glassdoor

Singapore

Onsite

Full Time

Senior

0 applicants

View Job Listing

Apply to 100+ jobs

About this role

The Software Engineer, Web Crawling at Exa is responsible for building and scaling a distributed web crawler to index the entire internet for a new search engine. This role involves designing systems that optimize crawl efficiency and politeness across numerous domains, tackling dynamic content and anti-bot measures, and developing algorithms for intelligent crawl scheduling. Candidates should have extensive experience in high-performance languages such as Rust or C++ and a strong interest in improving knowledge accessibility.

Skills

About Exa

exa.ai

Exa is a state-of-the-art AI search engine and web crawling platform that provides robust APIs for retrieving real-time and structured web data. With its deep research tools, SERP API, and customizable web crawling capabilities, Exa empowers developers and enterprises to integrate high-quality search functionality directly into their applications. The platform is designed to enhance AI workflows by delivering comprehensive, relevant information from across the web while ensuring data privacy and compliance. Trusted by industry leaders, Exa offers a scalable solution tailored for diverse applications and business needs.