Research Engineer, Benchmarks & Evaluation
$204k - $352k • New York, NY (HQ) • Full-time
Posted 18d ago
Job Location
New York, NY (HQ)
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
Ramp Labs is seeking a Research Engineer to own the company's benchmarking and evaluation efforts for AI models. This role involves transforming real-world financial workflows into benchmarks and reinforcement learning environments, designing evaluation metrics and graders, and analyzing model performance to guide post-training efforts. The goal is to ensure that models and agents improve meaningfully and safely before being deployed to the 70,000+ companies that use Ramp's financial infrastructure.
Responsibilities
- Build benchmarks from real financial workflows, including agent traces, customer edge cases, and expert labels, for long-horizon agents.
- Design and implement graders, including rubrics, programmatic checks, and LLM-as-judge, calibrated against human agreement.
- Own the benchmarking process, deciding on metrics, running suites across models and agent versions, and reporting results for decision-making.
- Translate evaluation results into reward functions and RL environments for post-training, identifying and preventing reward hacking and contamination.
- Publish external benchmarks and analyses detailing model performance on financial tasks.
Requirements
- Hands-on experience building LLM benchmarks or evaluations used for real-world decision-making.
- Strong Python and software engineering skills, ensuring reproducible and fast evaluation code.
- Proficiency in statistical analysis, including variance, confidence intervals, sample size, and inter-rater agreement.
- Working knowledge of post-training techniques (SFT, RL with verifiable rewards) and their relationship with evaluations.
- Comfort working with SQL and large datasets, utilizing production traces.
- A keen sense for identifying and measuring what is important.
Benefits
- Flexible PTO
- Centralized home-office equipment ordering
- Health and wellness stipend
- Budget for intra-office travel
- Weekly coffee stipend
- 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents (US)
- One Medical annual membership (US)
- 401(k), including employer match (US)
- Fertility HRA (up to $10,000 per year) (US)
- Parental leave: up to 16 weeks (birthing + bonding) or 8 weeks (bonding only) at 100% pay
- Pet insurance (US)
- In-office perks: lunch, snacks, drinks, and more (US)
- Relocation expense coverage to NYC or SF (if needed)
- Group medical, dental, and vision coverage through Sun Life (Canada)
- Life, AD&D, and disability coverage (Canada)
- Fertility drug coverage (up to $4,000 lifetime) (Canada)
- Group Retirement Plan with employer match (RRSP + DPSP) (Canada)
- Employee Assistance Program and virtual care through Lumino Health (Canada)
- Private medical insurance through Freedom Elite (UK)
- Virtual GP and at-home care via eMed x Livi (UK)
- Workplace pension through Penfold, with salary sacrifice option (UK)
