Research, Evals
$180k - $350k • San Francisco, California • FullTime
Posted 1y ago
Job Location
San Francisco, California
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Exa is an applied AI lab building a search engine unlike any seen before, powered by massive-scale infrastructure for web crawling, state-of-the-art embedding models, and high-performance vector databases. We aim to build perfect search over all the world's information, far beyond Google. The ML organization is central to this mission, training foundational models for search. We are seeking an ML Evals Engineer to design and build our evaluation stack, investigating how to evaluate search engines in an LLM world and creating a comprehensive, creative, and effective eval suite. Your work will directly influence research directions and shape the company's future.
Responsibilities
- Design and build the evaluation stack for Exa's search engine.
- Investigate methods for evaluating search engines in an LLM context.
- Build comprehensive, creative, and effective evaluation suites.
- Write a manifesto defining perfect search.
- Design and implement evaluation frameworks to probe search engine limits.
- Build scalable, reliable eval pipelines to track regressions, drift, and quality signals.
- Create golden datasets, synthetic benchmarks, agentic tasks, and real-world test suites.
- Partner with ML researchers, data engineers, infra engineers, and product teams to shape feedback loops.
Requirements
- Hands-on ML experience, including training, finetuning, or evaluating models (bonus for embeddings or LLMs).
- Strong engineering fundamentals with the ability to build reliable systems (e.g., Python, Rust, distributed pipelines, GPU/cluster jobs).
- Enjoy diving into data through building eval sets, inspecting edge cases, and designing measurement strategies.
Benefits
- Premium healthcare benefits (medical, dental, vision).
- Fertility benefits.
- 16 weeks of fully paid parental leave for all new parents.
- Monthly wellness stipend.