Eval360 - Error Analysis Engineer
$150k - $450k • Remote • Sunnyvale, CA • Full-time
Posted 4mo ago
Job Location
Sunnyvale, CA
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Engineer
About the job
The Institute of Foundation Models is seeking an Eval360 - Error Analysis Engineer to join their dedicated research lab focused on building, understanding, using, and risk-managing foundation models. This role will focus on improving and operating Eval360, a critical evaluation service that acts as a quality gate for AI models. You will be instrumental in understanding model failures, categorizing issues, and enhancing evaluation systems to detect and prevent problems before model releases.
You will collaborate with a cross-functional team of researchers, machine learning engineers, data scientists, and product managers throughout the software development lifecycle. The ideal candidate is comfortable working across the full stack, including front-end interfaces, back-end pipelines, data analysis, infrastructure, databases, dashboards, and APIs, bringing strong software engineering skills and analytical judgment to translate ambiguous model failures into structured insights.
Responsibilities
- Collaborate with researchers, engineers, and stakeholders to implement software solutions for Eval360 and model evaluation workflows.
- Build and improve Eval360 as a quality gate for model development, comparison, and release decisions.
- Perform deep error analysis on model outputs, identifying failure patterns, categorizing issues, tracing root causes, and proposing improvements to evaluation methodology.
- Develop tools, workflows, and dashboards for inspecting model failures, comparing model behavior, and understanding quality regressions.
- Design and implement client-side and server-side architecture for evaluation review systems, error analysis interfaces, reporting tools, and internal applications.
- Develop responsive, usable interfaces for error triage, annotation review, evaluation debugging, and model quality investigation.
- Build and maintain back-end services, APIs, data pipelines, and integrations for evaluation execution, results storage, analysis, and reporting.
- Test software to ensure responsiveness, correctness, reliability, and efficiency across evaluation workflows.
- Troubleshoot, debug, and upgrade evaluation systems, identifying issues in data processing, metrics, model output handling, job orchestration, and analysis tools.
- Create and maintain security, access control, and data protection settings for evaluation data and tooling.
- Write clear technical documentation for Eval360 systems, error taxonomies, workflows, debugging procedures, and user tools.
- Improve evaluation quality, model diagnostics, and failure-mode visibility by working with researchers and data scientists.
- Stay updated on new development tools, evaluation frameworks, model analysis methods, data quality techniques, and architectures relevant to AI evaluation.
- Contribute to the design of error taxonomies, evaluation rubrics, quality thresholds, regression detection methods, and model readiness criteria.
- Help ensure Eval360 produces reliable, interpretable, and actionable signals for model quality gates.
- Contribute to research publications, technical reports, internal knowledge sharing, and external presentations.
- Contribute to intellectual property and thought leadership in AI evaluation, error analysis, model quality measurement, and evaluation infrastructure.
- Perform other duties as reasonably directed by the line manager.
Requirements
- Bachelor's degree in Computer Science, Machine Learning, Data Science, Software Engineering, Statistics, or a related technical field.
- Master's or Ph.D. in Computer Science, Machine Learning, Artificial Intelligence, Data Science, or a related field is preferred.
- Proven experience as a Software Engineer, Full Stack Developer, Machine Learning Evaluation Engineer, Data Scientist, AI Engineer, or similar role.
- Experience building software systems for AI, machine learning, data analysis, evaluation, annotation, experimentation, or model monitoring.
- Experience working with AI algorithms and developing systems that accommodate AI-related requirements.
- Experience performing error analysis, model evaluation, data quality analysis, or failure-mode investigation for machine learning or language model systems.
- Experience developing internal applications, dashboards, review tools, or web-based workflows.