Senior Systems Software Engineer, GPU Fleet Telemetry
$152k - $288k • Remote • US, CA, Santa Clara
Posted 2d ago
Job Location
US, CA, Santa Clara
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
NVIDIA is seeking a Senior Software Engineer to join their DGX Cloud / Fleet Intelligence team. This role involves building agent-side systems to collect critical data like GPU health, host telemetry, inventory, attestation evidence, and BMC data across NVIDIA's extensive GPU fleet. The position offers an opportunity to contribute to open-source software used by a wide range of users, from developers to operators and customers working with NVIDIA GPU infrastructure. You will be part of a diverse and supportive environment focused on innovation and defining the next era of computing powered by AI and GPUs.
Responsibilities
- Design and develop Fleet Intelligence agent software for Linux hosts, bare metal systems, and Kubernetes GPU nodes.
- Build telemetry and health collection for NVIDIA GPUs, DCGM/NVML, drivers, CUDA runtime, InfiniBand, containers, kernel/OS state, CPU, memory, disk, and networking.
- Develop and maintain workflows for inventory, enrollment, node identity, local state, attestation, and backend export, including local API, Prometheus metrics, file export, and OTLP/HTTP export paths.
- Build and improve Kubernetes daemonset deployment, systemd service packaging, .deb/.rpm packaging, and container image workflows.
- Contribute to open-source Fleet Intelligence agent and collector software, including code, tests, documentation, release artifacts, and community-facing engineering practices.
- Contribute to out-of-band collection using Redfish/BMC interfaces for inventory, GPU attestation, BMC metrics, and logs.
- Improve collector concurrency, rate limiting, retry behavior, credential handling, partial-failure handling, and backend submission semantics.
- Collaborate with backend, infrastructure, SRE, security, and datacenter operations teams to ensure reliable GPU fleet observability.
Requirements
- 5+ years of industry software engineering experience with a Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience.
- Strong Go development experience for Linux services, CLIs, agents, or daemons.
- Rust experience, or strong willingness to work in Rust, for out-of-band collector development.
- Strong Linux systems knowledge, including processes, filesystems, networking, service lifecycle, logs, permissions, and host diagnostics.
- Experience with telemetry, observability, health monitoring, or fleet-management systems.
- Experience with Docker, Kubernetes, Helm, and production deployment workflows.
- Experience contributing to open-source projects or working in public repositories with code review, issue tracking, documentation, release notes, and signed commits.
- Familiarity with secure credential handling, enrollment flows, tokens/JWTs, and service-to-service authentication.
- Strong debugging skills across hardware-adjacent software, operating systems, containers, and distributed backend integrations.
Benefits
- Equity
- Benefits