Senior / Staff AI Engineer — Evals & Model Quality

About the job

Responsibilities

• Build evaluation frameworks and infrastructure to measure AI agent accuracy, quality, and reliability across various workflows.
• Develop golden datasets, benchmarks, and regression suites in collaboration with domain experts.
• Design evaluations for complex agent behaviors, including multi-step trajectories, tool use, and reasoning.
• Create automated grading systems using LLM-as-judge, deterministic graders, and human evaluation workflows.
• Integrate offline and online evaluations into development and release processes for confident shipping.
• Continuously benchmark frontier and open-weight models on metrics like accuracy, latency, cost, and reliability to inform model selection.
• Establish continuous improvement loops by converting production failures and feedback into new evaluation cases and quality enhancements.

Requirements

• 5+ years of software engineering experience with hands-on experience building production AI/ML systems.
• Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
• Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
• Proficient Python engineering skills for building reliable, production-grade systems.
• Ability to independently own ambiguous technical problems from research to production.
• Clear communication skills, both written and verbal.
• Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking is a plus.
• Experience with open-weight models, fine-tuning, post-training, or model routing is a plus.
• Background in B2B SaaS and 0-to-1 experience is a plus.
• Demonstrated drive, grit, and ownership.

Benefits

• Fully covered, best-in-class health, dental, and vision benefits
• Competitive compensation
• Meaningful stock options
• Company 401(k)
• Unlimited PTO
• Lunch and dinner onsite
• Team events, such as happy hours and off-sites
• Pre-tax commuter benefits