Tech Lead, Applied AI - Evals
$300k - $500k • San Francisco • Full-time
Posted 11d ago
About the job
Mercor is seeking a Tech Lead for its Evals team to own the end-to-end evals product. This role involves defining the product's architecture and roadmap, and staying hands-on with coding. The core challenges include reliably grading open-ended professional work, ensuring reproducible results with evolving models and tasks, and scaling long agent tasks. This position offers the opportunity to contribute to both research and engineering aspects, with potential to expand into post-training and new product areas, ultimately helping to improve models based on identified failures. Mercor's mission is to organize human intelligence for the AI economy, building infrastructure for AI labs and enterprises to train, evaluate, and improve AI on real-world professional work.
Responsibilities
- Own the evals flywheel, setting architecture and roadmap, and making tradeoff decisions.
- Lead the development of APIs, data models, and UI for model evaluation and comparison.
- Build systems to run new frontier models on APEX benchmarks and publish results to the public leaderboard.
- Develop failure analysis tools to pinpoint the exact steps where models fail.
- Calibrate graders against expert judgment and manage grader/agent updates.
- Ensure product reliability and manage compute costs.
- Engage directly with customers to gather feedback and inform the product roadmap.
- Hire and mentor engineers, and manage the team.
Requirements
- Bachelor's Degree in Computer Science or a related field.
- 5+ years of experience building and running backend systems in production, with senior or staff level experience.
- Strong full-stack skills in a modern backend language (Python, Go, or TypeScript) and a frontend framework like React.
- Proven track record of owning a platform or product area end-to-end, including team leadership.
- Strong product sense focused on user needs.
- Experience building data-heavy products with data models, APIs, and views over large result sets.
- Excellent written and verbal communication skills for interacting with engineers, researchers, and customers.
- Comfort with ambiguity and a habit of iterative shipping.
Benefits
- Generous equity grant vested over 4 years
- Health, dental, and vision insurance
- 401(k) with company match
- Unlimited paid vacation and sick days
- Parental leave for birthing and non-birthing parents
- Free mental health services
- $200 monthly wellness reimbursement
- $1.5K monthly meal stipend
- $200 monthly laundry reimbursement
- Free Equinox membership
- Up to $15K relocation bonus
