Product Manager, Public Sector GenAI Test & Evaluation (T&E)
$154k - $257k • Remote • San Francisco, CA; St. Louis, MO; New York, NY; Washington, DC
Posted 2mo ago
Job Location
San Francisco, CA; St. Louis, MO; New York, NY; Washington, DC
Tech Stack
Remote Work Policy
Fully remote
Categories
Applied AI Engineer
About the job
Scale is seeking a high-horsepower technical leader to join their Public Sector team as a GenAI Test & Evaluation (T&E) Product Manager. This role involves defining the vision and owning the roadmap for evaluation capabilities, crucial for measuring, improving, and proving the performance of agentic applications in high-stakes government environments. You will collaborate across engineering organizations to identify and resolve technical bottlenecks, driving execution and ensuring evaluation services meet the demanding requirements of government use cases. The primary objective is to refine the technology stack that supports ML team advancements and provides critical performance insights to stakeholders.
Responsibilities
- Define the vision and own the roadmap for AI evaluation capabilities.
- Manage the T&E tech stack, including infrastructure for measuring and improving AI application performance.
- Identify and address technical friction points across multiple engineering organizations.
- Drive the execution of actionable plans to improve evaluation processes.
- Define requirements for evaluation services to meet government use case demands.
- Refine the tech stack to enable ML teams to improve performance.
- Surface critical performance information to stakeholders.
- Provide consistent weekly reporting on delivery velocity and blockers to executive stakeholders using technical project management frameworks.
Requirements
- 3+ years of experience in software engineering, systems architecture, or highly technical program management.
- Ability to read code, understand system architecture, and participate in technical design reviews.
- Proven experience designing, owning the roadmap for, or operating AI application evaluation infrastructure.
- Demonstrated experience translating vague problems into technical roadmaps with measurable success metrics.
- Proven track record of successfully shipping projects from an undefined state in high-pressure environments.
- Experience leading projects requiring alignment between at least three distinct engineering organizations.
- Experience using technical project management frameworks (e.g., Linear) for reporting.
Benefits
- Base salary
- Equity
- Comprehensive health, dental, and vision coverage
- Retirement benefits
- Learning and development stipend
- Generous PTO
- Commuter stipend (potential)