Staff+ Software Engineer, ML Sampling Path

About the job

Responsibilities

• Design, build, and operate backend systems processing every token on Claude's generation path, including the streaming contract with the API and inference engines.
• Own latency and reliability end-to-end, defining and maintaining SLOs and error budgets for latency, time-to-first-token, and availability.
• Lead incident response and postmortem follow-through.
• Ship changes to the hot path rapidly but safely using canary and gradual rollouts, error budget and latency gating, and fast rollbacks.
• Drive per-token performance, chasing tail latency and keeping costs flat as traffic, models, and checks per request grow.
• Set technical direction for the sampling path, leading design reviews and making trade-off calls on latency, reliability, and cost with inference and research teams.
• Mentor engineers and raise the operational bar for the wider Safeguards organization.

Requirements

• Designed, built, and operated high QPS systems at global scale, with accountability for incident response, outages, and postmortem-driven remediation.
• Strong foundation in distributed systems, including replication, consistency tradeoffs, failure modes, and SLO management under load.
• Experience designing systems for graceful degradation, planning for slow dependencies, dropped streams, or partial rollouts.
• Successfully shipped broad or all-encompassing changes to mission-critical systems (e.g., database migrations, interface changes, rewrites).
• 8+ years of industry software engineering experience.
• Familiarity with LLM inference systems and transformer-based models (a plus).