Product Manager, APEX
San Francisco • FullTime
Posted 26d ago
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Mercor is seeking a Product Manager to lead and scale the APEX brand and public leaderboard. APEX assesses the effectiveness of frontier AI models in performing economically valuable work. In this role, you will define the positioning, strategy, roadmap, and operational excellence for Mercor's evaluation products, benchmarks, and leaderboards. You will collaborate with Research, Engineering, Operations, and Go-To-Market teams to transform evaluation datasets into trusted industry benchmarks that influence AI model development and purchasing decisions. You will serve as the product owner for APEX and the evaluation platform, driving innovation, integrity, infrastructure, customer adoption, and business impact. You will engage directly with frontier AI labs and enterprise customers, acting as a thought leader in AI evaluation and measurement, and partnering with Mercor's renowned benchmark research team.
Responsibilities
- Own the roadmap and portfolio strategy for new benchmark development, leaderboard launches, infrastructure investment, and expansion into new evaluation categories.
- Manage the intake process for new benchmarks, evaluating and prioritizing proposals based on demand and company strategy.
- Enforce evaluation integrity, including contamination policy, holdout strategy, versioning, auditability, and release cadence.
- Build and manage the end-to-end evaluation pipeline, from run execution to published results, including harness execution, grading, model onboarding, and reporting.
- Engage directly with AI labs and customers to understand evaluation needs, present results, and gather feedback for future benchmarks.
- Support Go-To-Market teams on launches, partnerships, and thought leadership initiatives.
- Connect leaderboard demand signals to investment in loss analysis and dataset production.
- Track adoption, usage, and downstream revenue to demonstrate leaderboard ROI.
- Write specifications and PRDs, review trajectories, identify failures, and contribute to system improvements.
Requirements
- 5+ years in product management, technical program management, or a customer-facing technical role.
- Prior background in Software Engineering, Machine Learning, or Data Science is strongly preferred.
- Fluent reasoning about rubric design, inter-rater reliability, agentic harnesses, contamination and overfitting.
- Ability to distinguish real capability gaps from measurement noise.
- Experience publishing neutral, precise, and defensible numbers under scrutiny.
- Strong instincts for identifying valuable capabilities to measure and significant results to highlight.
- Proven track record of building a product, program, or business line from inception.
- High sense of ownership and accountability for outcomes.
- Ability to self-direct in ambiguous contexts and create clarity for others.
- Skill in aligning stakeholders across research, engineering, operations, and Go-To-Market teams around a shared methodology.
Benefits
- Bi-annual performance bonus structure
- Generous equity grant vested over 4 years
- Up to $15k Relocation bonus
- $10K housing bonus (if living within 0.5 miles of office)
- $1.5K monthly stipend for meals
- Free Equinox membership
- $200 monthly laundry reimbursement
- $200 monthly personal wellness reimbursement
- Health, Dental, Vision insurance