Product Designer, Evals & Prompts
$50k - $100k • San Francisco, CA
Posted 9d ago
Remote Work Policy
On-site
Categories
Prompt Engineer
About the job
Anthropic is building reliable, interpretable, and steerable AI systems to be safe and beneficial for users and society. This role is a foundational member of the Product Prompt and Eval Design team, focusing on the 'evals' side of defining Claude's behavior. You will build the systems that check prompts, the harness that runs them, and the tools that empower designers to perform this work. This position involves close collaboration with surface owners, product teams, and prompt engineering teams to ensure Claude aligns with user expectations, product strategy, and safety requirements across all product launches.
Responsibilities
- Write and revise prompts for Claude's tools, features, and behaviors, testing surfaces and implementing prompt fixes.
- Build graders to automate prompt fix verification and rerun evaluations on new models.
- Develop visual, low-code eval tools for designers to create comparison sets, automate rubrics, and compare prompt variants.
- Observe and simplify eval tools based on designer usage.
- Support model releases by testing surfaces, writing prompt fixes and migrations, and creating prompts for new features.
- Establish and scale the eval harness to exercise tools with trustworthy settings and maintain eval green status across models.
- Package prompt-related issues for training, including graders for clear behaviors and human-feedback questions for nuanced ones.
Requirements
- Production-quality Python.
- Experience building and maintaining evaluation pipelines for LLM products (graders, rubrics, regression suites).
- Experience building internal tools with user-friendly interfaces for non-technical users.
- Experience setting up test harnesses, sandboxing tool calls, and pinning settings for comparable runs.
- Experience shipping prompts or working closely with prompt engineers to understand prompt behavior across models.
- Ability to analyze transcripts and identify evaluation shortcomings.
Benefits
- Annual compensation range: $305,000 - $385,000 USD
- Visa sponsorship available