Member of Technical Staff, Data & Evals
$230k - $340k • Remote • San Francisco • FullTime
Posted 2h ago
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Harvey is transforming legal and professional services by combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise. This role is an opportunity to join a fast-scaling company at an inflection point, defining a new category in real-time. The team operates with intensity, ownership, and a commitment to excellence, guided by values of Decisiveness, Simplicity, and Job's Not Finished. We are looking for driven individuals to do the best work of their careers alongside like-minded colleagues.
Responsibilities
- Build synthetic data pipelines for generating, filtering, and curating training and evaluation samples for complex legal reasoning and agentic workflows.
- Design evaluations and verifiers that accurately reflect real legal work, calibrate them against expert judgment, and investigate their failure points.
- Build and maintain capability benchmarks, such as the Legal Agent Benchmark (LAB), and product-focused evaluations to track model performance on customer workflows.
- Collaborate with legal domain experts to define quality metrics, rubrics, and grading mechanisms for training and evaluation.
- Ensure the quality of training and evaluation data, including provenance, coverage, and safeguards against contamination.
- Integrate product usage and customer feedback into new evaluations and training data to drive model improvements in areas critical to customers.
- Partner with researchers to analyze model failures, differentiate genuine performance gains from misleading metrics, and inform training and release decisions.
Requirements
- Experience building data pipelines, ML evaluation systems, or experimentation tooling.
- Strong Python skills and experience with production-quality code ownership.
- A rigorous approach to measurement, including statistical reasoning and experimental design.
- Ability to translate ambiguous goals or expert judgment into structured datasets, rubrics, and measurable criteria.
- Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus.
- Strong data and measurement engineering experience close to ML model development is valuable even without prior LLM work.