Site Reliability Engineer, Provider Operations
$245k - $295k • Remote • Remote (US) • FullTime
Posted 4d ago
Job Location
Remote (US)
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Engineer
About the job
OpenRouter is seeking its first AI Inference SRE to ensure the operational health of its provider supply. This role will focus on making sure every endpoint is fast, correct, and available, and that problems are detected and routed around before customers are impacted. The SRE will be responsible for building and owning monitoring systems, improving detection and failover mechanisms, and managing incident response for provider-related issues. This position involves holding providers accountable through telemetry and SLO reporting, detecting quality regressions, automating manual operational tasks, and building tooling for capacity planning and launch readiness.
Responsibilities
- Build and own monitoring for provider and endpoint health, including latency, throughput, error rates, uptime, and output correctness.
- Set SLOs per provider tier and alert on them.
- Improve detection speed for degraded endpoints and work with the routing team for automatic traffic shifting.
- Own on-call responsibilities for provider incidents, including triage, mitigation, communication with providers, and postmortems.
- Develop telemetry into scorecards and SLO reports for providers and act as a technical escalation point for misbehaving endpoints.
- Build continuous canaries and evaluations to detect silent regressions in LLM inference.
- Automate manual provider operations tasks such as disabling endpoints, capacity changes, and rate-limit tuning.
- Build tooling to load-test endpoints before major launches to prevent traffic-related failures.
Requirements
- 4+ years of experience in SRE, production engineering, or infrastructure roles managing high-traffic, customer-facing systems.
- Proficiency with observability tooling and practices, including metrics, tracing, logs, SLOs/error budgets, and actionable alerting.
- Skilled software engineer with a preference for writing tools over executing runbooks, using TypeScript and/or Python.
- Experience with distributed systems failure modes such as timeouts, retries, backpressure, partial outages, and noisy neighbors.
- Ability to act as a calm and clear incident commander, communicating effectively with external partners under pressure.
- Understanding of, or eagerness to deeply learn, LLM inference serving, including streaming, tool calling, prompt caching, throughput/latency tradeoffs, and provider API differences.