Senior Production Engineer - DGX Cloud
$184k - $357k • Remote • US, CA, Santa Clara
Posted 17d ago
Job Location
US, CA, Santa Clara
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
NVIDIA DGX Cloud provides AI services and endpoints for research and production workloads. We are seeking a Senior Production Engineer to develop software and automation that ensures these services are reliable, scalable, and safe to operate. The Production Engineering team focuses on large-scale distributed systems, including internal and external model endpoints, regional control plane services for workload orchestration and request routing, and the GPU/CPU compute infrastructure for inference and agentic workloads. Our work encompasses Kubernetes clusters across various cloud environments (AWS, Azure, Google Cloud, partner clouds) and on-premises deployments.
Responsibilities
- Build and operate production software, automation, and tooling for control plane services, model deployments, and inference/agentic workloads.
- Enhance the reliability of inference and agentic platforms and services, including NVIDIA Cloud Functions, SGLang- and vLLM-based endpoints, and NVIDIA Dynamo inference services.
- Improve endpoint availability, inference routing, capacity management, and service health to maintain predictable performance under changing workloads and demand.
- Utilize infrastructure as code and GitOps for consistent deployment, configuration, validation, upgrade, and recovery of services across environments.
- Develop workflows for service enablement, model releases, handoffs, deprecation, and ongoing operations, automating repeatable manual tasks.
- Define and instrument SLIs and SLOs for inference and control plane services, using error budgets to drive reliability improvements and make production health visible.
- Participate in on-call and incident response, troubleshoot failures across various system layers, and convert recurring issues into automation and durable fixes.
- Collaborate with cross-functional teams (model, platform, storage, networking, security, GPU infrastructure) to design and operate services safely at scale.
Requirements
- 8+ years of experience building or operating production services and large-scale distributed systems, including hands-on automation.
- Strong programming skills in Python, Go, or a comparable language, with experience developing production operations tools.
- Experience with infrastructure as code, configuration management, or GitOps, and building automation for repeatable service deployments.
- Strong knowledge of Linux, Kubernetes, containers, cloud infrastructure, distributed systems, and networking fundamentals.
- Understanding of SRE principles, including SLIs, SLOs, error budgets, incident response, and reducing operational toil.
- Experience instrumenting services and using metrics, logs, and traces to understand system behavior and improve reliability.
- Clear technical communication skills and ability to collaborate across engineering teams.
- BS/MS in Computer Science or equivalent experience.
Benefits
- Equity
- Benefits