Data Annotation Lead
San Francisco • FullTime
Posted 2mo ago
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Physical Intelligence is seeking a Data Annotation Lead to manage and scale annotation operations and the team responsible for it. As annotation is crucial for model improvement and demand is rapidly increasing, this role will focus on expanding the annotation workforce from hundreds to thousands while simultaneously enhancing quality. You will be responsible for designing the organizational structure, training pipelines, quality systems, and key metrics to ensure efficient scaling. This position involves owning the people and the operation, encompassing throughput, quality, cost, and delivery across all annotation types.
Responsibilities
- Own annotation operations end-to-end, including throughput, quality, cost, and on-time delivery.
- Scale the annotation workforce from hundreds to thousands through workforce planning, org design, and managing the hiring and onboarding funnel.
- Build and lead a multi-layer management structure, hiring, developing, and managing managers and team leads.
- Increase throughput using autolabeling and model-based annotation by designing human-in-the-loop workflows.
- Establish and maintain a training and certification pipeline for new annotators and teams.
- Define and continuously improve the quality bar through rubrics, calibration, audit/QA loops, and quality-adjusted productivity.
- Implement operational metrics and reporting, driving week-over-week improvement.
- Manage capacity planning and prioritization, allocating teams to high-impact work.
- Oversee performance management with clear standards, feedback, and an improvement/exit process.
- Collaborate with product and engineering to define annotation tooling that enhances throughput and quality.
- Partner with research and project leads to translate annotation needs into clear instructions and SLAs.
- Manage the in-house vs. vendor mix and external partners.
- Own the annotation operating budget and unit economics, optimizing cost-per-annotation while maintaining quality.
Requirements
- 7+ years of experience leading scaled data or annotation operations, including managing teams of 100+.
- 3+ years of experience as a manager of managers.
- Proven track record of establishing new annotation programs from scratch.
- Deep understanding of annotation best practices, operations, and strategy.
- Experience integrating autolabeling and model-based annotation into human workflows.
- Proficiency with operational and quality metrics for data-driven management of large workforces.
- Strong cross-functional partnership skills with product, engineering, and research/ML teams.
- Clear written and verbal communication skills, capable of setting and enforcing standards across a large, distributed team.
- Working understanding of Machine Learning and the impact of annotation quality on model performance.