Research Crawling Engineer

ABOUT THE ROLE Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments. WHAT YOU'LL DO - Build and maintain web crawlers across social, travel, and multilingual websites. - Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day. - Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques. - Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale. - Collaborate with research teams to create datasets suited to model training and research needs. - Monitor crawl coverage, performance, and data quality, adapting systems as websites change. - Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments. WHAT WE'RE LOOKING FOR - At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD. - Experience operat... Click Apply to read the full job description.