Site Reliability Engineer
Southaven, MS; Memphis, TN
Posted 10h ago
Job Location
Southaven, MS; Memphis, TN
Tech Stack
Remote Work Policy
On-site
Categories
Machine Learning Engineer
About the job
SpaceX is building Super Intelligence to help humanity understand the universe, developing the full stack from supercomputers to AI models and applications. As a Site Reliability Engineer focused on campus reliability, you will be responsible for designing and ensuring the reliability of the campus infrastructure. This role involves technically commanding cross-discipline SEVs, building guardrails to minimize future incidents, and acting as the connective tissue across compute, network, storage, power, and cooling on one of the world's largest AI training campuses. The position requires calm incident leadership, fleet-scale observability judgment, and the ability to drive reliability improvements across both software and facility domains. Ideal candidates will have experience in power plants, nuclear power plants, data centers, or similar facilities with expertise in power, compute, cooling, and related plant systems.
Responsibilities
- Own monitoring architecture and signal quality for GPU, host, and facility telemetry, including alert suppression and trust.
- Provide SEV command support, including technical incident leadership, bridge coordination, paging, escalation, and timeline/severity management.
- Run blameless postmortems and drive corrective actions to completion.
- Gate high-risk changes, including fleet-scale firmware rollouts and work on power and cooling systems, through a disciplined change management process.
- Lead cross-functional reliability projects across compute, network, storage, and facility boundaries, addressing complex failures.
- Partner with Hardware Failure Analysis to address systemic failure classes by translating fleet data into monitoring and RCA inputs.
- Build and maintain playbooks, run game days, and keep dependency maps current, jointly owning runbook quality with the NOC.
- Define error budgets and availability objectives at campus and service boundaries.
- Participate in on-call rotations and incident response for SEV-class events.
Requirements
- Experience in power plants, nuclear power plants, data centers, or similar facilities spanning power, compute, cooling, or related plant systems.
- Proven large-scale incident command experience and calm technical leadership.
- Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality.
- Cross-team facilitation and systems engineering depth across software and facility boundaries.
- Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or equivalent experience.
- 5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production operations.
- Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
- Experience working across at least two of: compute, network, storage, power, and cooling/facilities telemetry.
- Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations partner.
- Proficiency in scripting (Python, Bash) and general experience in at least one systems language (C, C++, Java, Go, Rust, or similar).
- Excellent problem-solving skills with a data-driven approach.