New: upload your resume and get matched to AI jobs. Free to start →

Staff+ Software Engineer, Safeguards Evals

$320k - $485k • New York City, NY; San Francisco, CA; Seattle, WA • Full-time

Posted 11d ago

Job Location

New York City, NY; San Francisco, CA; Seattle, WA

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Visa Sponsorship

The posting says they sponsor visas

Categories

AI Engineer

About the job

Anthropic is building reliable, interpretable, and steerable AI systems to be safe and beneficial for society. This role is crucial for ensuring the safety of our AI models and the systems that govern them. You will design and implement evaluation methods, build datasets that represent real-world misuse, and integrate these into production pipelines that influence model training, agent changes, and launch decisions. This position sits at the intersection of applied ML research and engineering, focusing on improving how we evaluate both model behavior and agentic systems.

Responsibilities

  • Design and execute experiments to enhance evaluation quality, including developing methods for generating representative test data and simulating realistic user behavior.
  • Research the impact of various factors (e.g., multi-turn conversations, tool use, long context, user diversity) on model safety behavior.
  • Analyze evaluation coverage to identify measurement gaps and evolve evaluation methods to remain effective as model and agent capabilities advance.
  • Build and manage the evaluation harness for an agentic investigation system, defining metrics and test cases for complex, long-horizon agents.
  • Measure agent performance end-to-end, focusing on detection precision, recall, investigation quality, and robustness.
  • Construct RL environments to improve Claude's safety investigation capabilities.
  • Create high-quality evaluation datasets representing real-world misuse across various harm areas, drawing from real traffic patterns and synthetic generation.
  • Translate observed harm patterns into measurable evaluations in collaboration with Policy and Enforcement teams.
  • Ship successful research into evaluation, regression, and release pipelines that run during model training and agent changes.
  • Develop tooling to enable policy experts to author, run, and iterate on evaluations independently.
  • Surface evaluation findings to research and training teams to drive upstream model improvements.

Requirements

  • Experience building and maintaining data pipelines.
  • Working understanding of LLMs, their capabilities, and failure modes, particularly agentic systems with tool use and multi-step reasoning.
  • Strong data analysis skills with the ability to derive reliable insights from large datasets.
  • Ability to transition between research prototyping and production-quality code.
  • Capacity to translate ambiguous problems into concrete, testable experiments.
  • Deep commitment to AI safety and a desire to make a real impact.
  • 8+ years of industry software engineering or ML engineering experience (preferred).
  • Expertise in building or contributing to LLM or agent evaluation frameworks, benchmarks, or automated grading systems (preferred).
  • Extensive experience in trust and safety, content moderation, or abuse detection systems (preferred).
  • Experience in red teaming, adversarial testing, or jailbreak research on AI systems (preferred).
  • Experience with synthetic data generation or data augmentation (preferred).
  • Experience with distributed systems or large-scale data processing (preferred).
  • Experience with prompt engineering or building LLM-powered applications (preferred).

Benefits

  • Annual Salary: $320,000—$485,000 USD
  • Visa sponsorship available