Research Engineer / Scientist, Data & Evals
$230k - $340k • Remote • Remote • FullTime
Posted 10h ago
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Harvey is transforming legal and professional services by combining agentic AI, an enterprise-grade platform, and deep domain expertise. This is an opportunity to join a fast-scaling company at an inflection point, defining a new category in real time. The team operates with intensity, ownership, and a commitment to excellence, driven by values of Decisiveness, Simplicity, and Job's Not Finished. As a Member of Technical Staff in Harvey Research, you will build the data and measurement systems crucial for post-training, benchmarking, and agent capability development. You will own synthetic data pipelines, evaluation design, and systems that convert expert feedback into training signals, tackling the core research challenge of defining and measuring quality in tasks where correctness is not always automatically verifiable.
Responsibilities
- Build synthetic data pipelines for generating, filtering, and curating training and evaluation samples for complex legal reasoning and agentic workflows.
- Design evaluations and verifiers that reflect real legal work, calibrate them against expert judgment, and investigate their failure points.
- Build and maintain capability benchmarks, such as the Legal Agent Benchmark (LAB), and product-focused evaluations tracking model performance on customer workflows.
- Collaborate with legal domain experts to define quality metrics, rubrics, and grading mechanisms for training and evaluation.
- Ensure training and evaluation data quality, including provenance, coverage, and safeguards against contamination.
- Integrate product usage and customer feedback into new evaluations and training data to drive model improvements.
- Partner with researchers to investigate model failures, differentiate real gains from misleading metrics, and guide training and release decisions.
Requirements
- Experience building data pipelines, ML evaluation systems, or experimentation tooling.
- Strong Python skills and experience owning production-quality code.
- Rigorous approach to measurement, including statistical reasoning and experimental design.
- Ability to translate ambiguous goals or expert judgment into structured datasets, rubrics, and measurable criteria.
- Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus.
- Strong data and measurement engineering experience close to ML model development is valuable.