AI Infrastructure Engineer
$200k - $400k • San Francisco • FullTime
Posted 6h ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Mercor is seeking an AI Infrastructure Engineer to build and manage the GPU and accelerator platform that powers our AI workloads. This role involves overseeing clusters, schedulers, training and inference serving, and the observability and cost data associated with fleet usage. You will write production code daily and own systems end-to-end, from design to on-call responsibilities. Mercor is a leading AI data company, building the layer between human expertise and frontier models, with a mission to organize human intelligence to power the AI economy. We are a profitable Series C company valued at $10 billion.
Responsibilities
- Design, deploy, and operate GPU clusters across multiple cloud providers and on-premise data centers, including custom Kubernetes and Slurm operators.
- Build scheduling systems for efficient accelerator utilization, including gang scheduling, bin-packing, and fair-share quotas.
- Run distributed training infrastructure using frameworks like PyTorch Distributed, Ray, or Megatron, ensuring fast checkpointing and model loading.
- Deploy and tune high-throughput LLM serving stacks using vLLM or similar technologies, optimizing for parallelism, quantization, and performance metrics.
- Develop benchmarking and traffic-replay tools to measure key performance indicators like TTFT and TPOT against baselines.
- Create self-healing health-check and remediation systems to manage hardware, network, and software faults automatically.
- Instrument fleet utilization using tools like DCGM, Prometheus, and Grafana, and establish relevant recording rules and alerts.
- Build an end-to-end cost-attribution pipeline to track GPU-hour spend by team and workload, supporting capacity planning.
- Drive autoscaling, rightsizing, and idle-capacity recovery programs to optimize costs.
- Write clear runbooks and documentation, and communicate infrastructure tradeoffs to various stakeholders.
- Develop internal tools and dashboards for researchers and non-engineers.
Requirements
- 5+ years of experience building and operating production distributed systems or ML infrastructure (7+ for Staff level).
- Proficiency in production-quality Python, and Go or C++, with strong shell scripting skills.
- Deep hands-on experience with Kubernetes and containerization, plus working knowledge of Slurm or another HPC resource manager.
- Experience running GPU workloads at scale, including scheduling, orchestration, and troubleshooting.
- Hands-on experience with at least one distributed training or serving stack (e.g., PyTorch Distributed, Ray, vLLM, Megatron-LM).
- Deep experience with at least one major cloud provider.
- Familiarity with observability tools like DataDog, Prometheus, PromQL, and Grafana, and comfort with on-call duties for critical systems.
- Ability to own projects end-to-end and gather requirements with minimal direction.
Benefits
- Bi-annual performance bonus structure
- Generous equity grant vested over 4 years
- Up to $15k Relocation bonus
- $10K housing bonus (if living within 0.5 miles of office)
- $1.5K monthly stipend for meals
- Free Equinox membership
- $200 monthly laundry reimbursement
- $200 monthly personal wellness reimbursement
- Health, Dental, Vision insurance