New: upload your resume and get matched to AI jobs. Free to start →

Research Engineer, Benchmarks & Evaluation

$204k - $352k • New York, NY (HQ) • Full-time

Posted 18d ago

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Engineer

About the job

Ramp Labs is seeking a Research Engineer to own the company's benchmarking and evaluation efforts for AI models. This role involves transforming real-world financial workflows into benchmarks and reinforcement learning environments, designing evaluation metrics and graders, and analyzing model performance to guide post-training efforts. The goal is to ensure that models and agents improve meaningfully and safely before being deployed to the 70,000+ companies that use Ramp's financial infrastructure.

Responsibilities

  • Build benchmarks from real financial workflows, including agent traces, customer edge cases, and expert labels, for long-horizon agents.
  • Design and implement graders, including rubrics, programmatic checks, and LLM-as-judge, calibrated against human agreement.
  • Own the benchmarking process, deciding on metrics, running suites across models and agent versions, and reporting results for decision-making.
  • Translate evaluation results into reward functions and RL environments for post-training, identifying and preventing reward hacking and contamination.
  • Publish external benchmarks and analyses detailing model performance on financial tasks.

Requirements

  • Hands-on experience building LLM benchmarks or evaluations used for real-world decision-making.
  • Strong Python and software engineering skills, ensuring reproducible and fast evaluation code.
  • Proficiency in statistical analysis, including variance, confidence intervals, sample size, and inter-rater agreement.
  • Working knowledge of post-training techniques (SFT, RL with verifiable rewards) and their relationship with evaluations.
  • Comfort working with SQL and large datasets, utilizing production traces.
  • A keen sense for identifying and measuring what is important.

Benefits

  • Flexible PTO
  • Centralized home-office equipment ordering
  • Health and wellness stipend
  • Budget for intra-office travel
  • Weekly coffee stipend
  • 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents (US)
  • One Medical annual membership (US)
  • 401(k), including employer match (US)
  • Fertility HRA (up to $10,000 per year) (US)
  • Parental leave: up to 16 weeks (birthing + bonding) or 8 weeks (bonding only) at 100% pay
  • Pet insurance (US)
  • In-office perks: lunch, snacks, drinks, and more (US)
  • Relocation expense coverage to NYC or SF (if needed)
  • Group medical, dental, and vision coverage through Sun Life (Canada)
  • Life, AD&D, and disability coverage (Canada)
  • Fertility drug coverage (up to $4,000 lifetime) (Canada)
  • Group Retirement Plan with employer match (RRSP + DPSP) (Canada)
  • Employee Assistance Program and virtual care through Lumino Health (Canada)
  • Private medical insurance through Freedom Elite (UK)
  • Virtual GP and at-home care via eMed x Livi (UK)
  • Workplace pension through Penfold, with salary sacrifice option (UK)