Research Engineer, Evals

About the job

Responsibilities

• Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows.
• Design and run offline and online evaluations that measure model performance on real customer tasks.
• Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling.
• Analyze model and agent failures and translate them into improved evaluations, datasets, and training loops.
• Build reusable evaluation tools and quality building blocks for various product surfaces and workflows.
• Collaborate with research, engineering, product, and design teams to enhance system quality through experimentation.
• Foster a culture of scientific experimentation, clear measurement, and continuous iteration.
• Advance the methods for evaluating AI systems in regulated, adversarial, and high-consequence environments.

Requirements

• Experience training, evaluating, or improving modern ML systems.
• Strong programming skills and comfort working in research-heavy codebases.
• Experience building benchmarks, datasets, evaluation pipelines, or quality systems.
• Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas.
• Ability to design clean experiments and draw reliable conclusions from noisy results.
• Strong engineering judgment and a bias toward building.
• Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains.

Benefits

• Competitive salary
• Meaningful equity
• Platinum-level medical, dental, and vision insurance
• Unlimited PTO, sick leave, and parental leave
• Up to $100 per month reimbursement for personal health and wellness expenses
• 401(k) plan