Software Engineer, Applied AI- Evals
$220k - $250k • San Francisco • Full-time
Posted 11d ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
Mercor is building the infrastructure for the AI economy, enabling AI labs and enterprises to train, evaluate, and improve AI models. The company's evals platform helps customers understand model performance on real-world tasks, identify failure points, and make informed decisions about model development and deployment. This role is at the intersection of frontier AI, human expertise, and real-world application, offering the chance to influence the capabilities of advanced AI systems. Mercor is a profitable, well-funded company with a strong emphasis on collaboration and high standards.
Responsibilities
- Build the product customers use to evaluate models, including APIs, data models, and UI for running evals, comparing models, and sharing scores.
- Own APEX launches by running new frontier models, publishing results to the public leaderboard, and ensuring accuracy and defensibility.
- Develop failure analysis tools to help customers understand where their models fail and identify specific steps where issues occurred.
- Build dataset views, registries, and versioning to make benchmarks easy to find and manage, and allow customers to bring their own datasets.
- Develop onboarding, roles, key management, and usage views to enable new teams to self-serve with the evals platform.
- Work directly with customers in pilots to gather feedback and inform the product roadmap.
- Contribute to post-training and new product areas, turning identified failures into data that improves models.
Requirements
- Bachelor's Degree in Computer Science or a related field.
- Strong full-stack skills in a modern backend language (Python, Go, or TypeScript) and a frontend framework (e.g., React).
- Proven track record of owning product features end-to-end, from concept to production.
- Strong product sense with a focus on user needs.
- Experience building data-heavy products, including data models, APIs, and views for large result sets.
- Excellent written and verbal communication skills for interacting with engineers, researchers, and customers.
- Comfort with ambiguity and a habit of iterative shipping.
- Experience with LLM evals, benchmarks, or AI developer tools is a plus.
- Startup experience or experience building B2B developer products is a plus.
Benefits
- Generous equity grant vested over 4 years
- Health, dental, and vision insurance
- 401(k) with company match
- Unlimited paid vacation and sick days
- Parental leave for birthing and non-birthing parents
- Free mental health services
- $200 monthly wellness reimbursement
- $1.5K monthly meal stipend
- $200 monthly laundry reimbursement
- Free Equinox membership
- Up to $15K relocation bonus
