Senior | Staff Software Engineer - AI / ML

About the job

Responsibilities

• Reduce the cost of long-horizon agent evaluation through adaptive sampling, early stopping, model cascades, caching, and cost-effective gating.
• Optimize AI model routing to select the cheapest model that meets quality standards, including fallback, monitoring, and cost attribution.
• Fine-tune and serve open-weight models (e.g., LoRA) when they achieve frontier quality, and identify when they do not.
• Develop models to predict task difficulty for frontier systems before execution.
• Build golden datasets and quantify the accuracy and calibration of LLM-as-judge systems for reproducible quality.
• Transform research prototypes into reusable, configurable components for engineers and researchers.

Requirements

• 5+ years of experience building production ML or software systems with end-to-end ownership.
• Hands-on experience running LLM or ML workloads in production and reasoning about non-deterministic systems.
• Deep understanding of statistics and experimentation, including experiment design, hypothesis testing, sampling, and confidence intervals.
• Strong Python and software engineering fundamentals, including testing, code review, and system design.
• Experience designing rigorous evaluations and interpreting results.
• Proactive in identifying high-impact problems and communicating effectively with researchers, engineers, and business partners.

Benefits

• Opportunity to work on frontier AI problems.
• Focus on ML and LLM challenges without extensive infrastructure work.
• Founding impact on the ML engineering discipline and team growth.
• Visible results impacting the speed, quality, and cost of AI data.
• Meaningful opportunities to shape priorities and strategic decisions.
• Support for career growth, learning, and skill development.