Member of Technical Staff, Enterprise Evals Platform
San Francisco • FullTime
Posted 1mo ago
Job Location
San Francisco
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Mercor is seeking a Member of Technical Staff for their Enterprise Evals Platform. This role is crucial in ensuring the reliability and economic viability of complex enterprise agent systems. You will be responsible for devising new methods for evaluation and rubrics, building upon Mercor's experience with domain experts and benchmarks. The core of this platform engineering role involves building the verifiers, environments for agent measurement, and scalable grading infrastructure. Your work will abstract across customers, domains, and tasks, ensuring each agent run contributes to the improvement of future agents, rather than starting from scratch.
Responsibilities
- Define golden sets by decomposing real tasks and encoding expert quality standards.
- Build robust verifiers for agent trajectories and outputs that are calibrated and difficult to game.
- Develop the evaluation platform to run offline environments, task suites, and grading at scale.
- Analyze production trajectory loss and convert failure modes into regression tests.
- Manage the optimization loop across models, prompts, skills, and harnesses.
- Oversee rollout gates for agent changes.
- Collaborate with the Enterprise Platform team and Applied AI engineers.
Requirements
- Professional, academic, or research experience in agent engineering and evaluation, including understanding agent runtimes, harnesses, trajectories, and failure points.
- Experience building evaluation suites for LLM or agent systems.
- Familiarity with benchmarks like terminal-bench, tau-bench, and APEX, including how they are constructed and gamed.
- Strong judgment in task and rubric design, capable of translating qualitative notions of quality into measurable metrics.
- Demonstrated improvements in agent or model performance based on evaluation work.
- Solid software engineering fundamentals.
- Ability to work independently on ambiguous, loosely specified problems.
- Bonus: Experience with Harbor environments and RL environments.
Benefits
- Up to $15k relocation bonus
- $10K housing bonus (if living within 0.5 miles of office)
- $1.5K monthly stipend for meals
- Generous equity grant vested over 4 years
- Free Equinox membership
- $200 monthly laundry reimbursement
- $200 monthly personal wellness reimbursement
- Health, Dental, Vision insurance