Staff ML Engineer, Agent Training & Environments
$250k - $280k • Remote • San Francisco Bay Area
Posted 1mo ago
Job Location
San Francisco Bay Area
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Agent Engineer
About the job
Labelbox is building the critical infrastructure for breakthrough AI models, focusing on data-centric approaches. We offer integrated solutions for frontier AI development, including an Enterprise Platform & Tools for scalable data annotation, a Frontier Data Labeling Service with specialized expertise, and an Expert Marketplace connecting AI teams with skilled annotators. We operate in a high-impact, fast-paced environment that rewards ownership, technical excellence, and continuous learning. This role is central to advancing frontier agent capabilities, bridging the gap between training and infrastructure by building the systems that run experiments, including environments, verifiers, and fine-tuning pipelines. We seek engineers with strong judgment who can set technical direction, build reliable systems quickly, and are at the forefront of agent-first engineering practices.
Responsibilities
- Develop RL environments for agentic tasks, including task definitions, tool surfaces, state and reset semantics, and reward design.
- Build harnesses to run thousands of RL environments in parallel.
- Create verifiers and graders, including programmatic checks, LLM judges, rubric pipelines, and pass@k scoring, to ensure trustworthy agent success judgments at scale.
- Implement fine-tuning pipelines for SFT and RL, covering data collection, training, and checkpoint evaluation, to translate evaluation signals into measurable agent improvements.
- Design and build evaluation systems to measure model and product quality by running millions of agent trajectories.
- Develop and scale training and serving infrastructure to meet the throughput needs of frontier labs, including multi-launcher orchestration, long-running job fault tolerance, and cost accounting.
Requirements
- 3+ years of experience shipping reliable systems relied upon by customers and engineers.
- Demonstrated ability to ship high volumes of code with exceptional quality.
- Strong judgment in system and API design, with the ability to make and defend architectural decisions.
- Experience shipping production code with coding agents daily and understanding their reliability requirements.
- Experience building foundational infrastructure like tooling, CI, harnesses, and libraries.
- Ability to move fast in ambiguous, startup-pace environments with influence over authority.
- Deep proficiency in Python and comfort across the rest of the technology stack.
- Experience fine-tuning models for agentic tasks using SFT and at least one RL method (GRPO, PPO, DPO, or similar) in production.
- Experience building environments for agents and understanding the challenges of reward and task design.
- Experience designing verifiers or graders for open-ended work and understanding potential gaming scenarios.
- Proficiency in forensically debugging training runs.
- Ability to reason about compute economics and optimize for cost-effectiveness.
- Experience writing up learnings to influence team practices.