Principal Engineer - Data Retrieval

About the job

Responsibilities

• Design scalable frontier storage and queueing systems with high-throughput writes, efficient lookups, deduplication, and scheduling based on data quality, diversity, and freshness.
• Build a continuous discovery loop that extracts links and feeds new candidates back into the crawl frontier.
• Develop a shared crawl execution layer that fetches reliably and politely at scale, handling dynamic websites while keeping resource use under control.
• Build extensible pipelines for open-web, licensed, and partner content, including text, PDFs, code repositories, and multimodal assets.
• Define stable interfaces for source connectors to allow teams to add sources without rebuilding the platform.
• Partner with parsing and curation teams to preserve valuable code, math, and structured content, and improve accepted data yield.
• Implement provenance and policy enforcement with legal, privacy, security, and data governance partners, translating approved publisher preferences, opt-outs, licensing terms, and use restrictions into auditable controls.
• Own production readiness, including capacity planning, observability, fault tolerance, recovery, and cost efficiency.
• Instrument the pipeline end to end and set service objectives for throughput, freshness, reliability, and meeting training cutoffs.
• Work with research and product teams to evaluate internal crawls and external datasets through comparable pilots.
• Deliver reproducible data releases that tie source coverage and processing quality to controlled training experiments.
• Coordinate dependencies across teams and make clear tradeoffs between immediate data delivery and long-term platform development.

Requirements

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience.
• 15+ years of relevant experience in software engineering, including designing and operating large-scale distributed systems, setting technical direction, and delivering complex infrastructure across multiple teams.
• Deep experience with web crawling or closely related discovery, indexing, or ingestion infrastructure, including distributed scheduling, stateful services, storage, and high-throughput data processing.
• Strong hands-on engineering and system design skills, with the ability to move between architecture, implementation, performance analysis, and production debugging.
• Experience building batch or streaming pipelines and reasoning about consistency, retries, duplicate processing, backpressure, and recovery.
• A track record of improving throughput, latency, resource efficiency, or operational reliability.
• Experience delivering data platforms for research, ML, search, or other demanding technical users.
• Ability to translate evolving requirements into clear interfaces and measurable engineering outcomes.
• Strong technical communication and leadership through influence across engineering, product, research, and governance teams.

Benefits

• Equity
• Benefits