Sr Staff Site Reliability Engineer, AI Infrastructure
$175k - $265k • Santa Clara • FullTime
Posted 18d ago
About the job
d-Matrix is seeking a Sr Staff Site Reliability Engineer to join their AI Infrastructure team. This role is crucial for owning the infrastructure layer that powers all engineering teams and customers, encompassing colocation facilities, on-premises GPU clusters, cloud environments, and platform services. You will be responsible for reliability, automation, and observability across these diverse environments, managing systems end-to-end from provisioning to incident response. This is a hands-on position focused on building and operating real infrastructure, collaborating with hardware and software teams on CI/CD, QA, and HPC workloads, and supporting customer-facing deployments.
Responsibilities
- Own reliability and availability across colo server fleets, on-premises lab clusters, cloud environments (AWS, Azure, GCP), and customer-facing platform services.
- Perform hands-on infrastructure work including server provisioning, OS configuration, networking, storage, and hardware troubleshooting from bare metal through auto-scaling Kubernetes environments.
- Lead capacity planning and hardware lifecycle management, and track cloud spend for FinOps and workload placement decisions.
- Drive all provisioning, deployment, and operational changes using Terraform and/or Ansible, and contribute to shared IaC modules.
- Build and document automation to eliminate toil, including host lifecycle management, fleet health checks, auto-remediation, self-service tooling, and networking automation.
- Design and maintain monitoring, alerting, and SLIs (Prometheus/Grafana, DataDog, Splunk, or equivalent), contributing to AIOps-driven detection workflows.
- Participate in on-call rotation, triaging and resolving incidents, and producing high-quality RCAs for critical incidents.
- Support platform services for internal teams and external customers, ensuring QoS and uptime commitments, and documenting operational runbooks.
Requirements
- Bachelor's or Master's in Computer Science, Electrical Engineering, or related field (or equivalent experience).
- 7+ years in SRE, infrastructure engineering, or systems administration.
- Strong Linux systems knowledge with hands-on colocation or on-premises server infrastructure experience (networking, storage, systemd, kernel parameters, performance diagnostics, physical hardware, rack networking, bare-metal provisioning).
- Production IaC experience with Terraform and/or Ansible (writing and maintaining configurations).
- Kubernetes operational experience (cluster troubleshooting, workload management, storage, networking).
- Experience with observability tooling (Prometheus/Grafana, DataDog, Splunk, or equivalent), including building dashboards and writing alert rules.
- Production-quality Python and/or Bash scripting.
- Incident response experience (structured triage, RCA production, follow-through on action items).