Senior / Staff ML Ops Engineer
Salary not listed • Remote • Remote US & Canada • Full-time
Posted 1mo ago
No salary is listed. Many US states require employers to post a pay range, so ask about pay early.
Job Location
Remote US & Canada
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
Waabi is a leader in Physical AI, developing technology for autonomous transportation. We are seeking a Senior/Staff ML Ops Engineer to build and evolve our training infrastructure on Kubernetes, focusing on GPU scheduling, autoscaling, and multi-node distributed jobs. You will shape the developer-facing surface of our tools, shorten the development loop for ML engineers, and evangelize best-in-class tooling. This role involves strengthening our data and artifact layer, turning one-off scripts into durable, tested libraries, and making experiments legible through robust experiment tracking and model registries. You will also ship CI/CD for models, build comprehensive observability across the ML stack, and treat documentation and onboarding as a core product surface. Driving adoption through user empathy and prototyping is key, alongside ensuring platform reliability and building secure guardrails.
Responsibilities
- Build and evolve training infrastructure on Kubernetes, including GPU scheduling, autoscaling, and distributed jobs.
- Shape developer-facing tools like CLIs and SDKs to simplify job submission and iteration.
- Reduce the time to first training run and edit-to-signal latency.
- Evaluate and advocate for ML tooling and frameworks.
- Enhance dataset versioning, sharding, and high-throughput loading for multimodal sensor data.
- Develop durable, tested, and observable Python libraries, CLIs, and services.
- Implement experiment hygiene, dashboards, model registries, and lineage tracking.
- Ship CI/CD for models, validating changes similarly to code changes.
- Build observability for the ML stack, including utilization, throughput, and cost.
- Provide excellent documentation, onboarding, and support for ML infrastructure.
- Drive adoption of platform tools through user prototyping and iteration.
- Ensure platform reliability with fewer failures and faster recovery.
- Collaborate with Security, IT, and Infrastructure to implement access controls and cost governance.
Requirements
- 5+ years of software or infrastructure engineering experience, including tools/platforms for other engineers and operating ML/data-intensive production systems.
- Hands-on Kubernetes expertise, including GPU scheduling, autoscaling, and networking fundamentals.
- Excellent Python skills with a track record of designing user-friendly APIs and CLIs.
- Practical AWS depth, including object storage, IAM, GPU compute, networking, cost management, and infrastructure as code (Terraform, Pulumi).
- Experience with distributed training in PyTorch (DDP, FSDP) and experiment tracking/model registry tooling.
- Fluency with containers, CI/CD, and modern build systems, including large monorepos.
- Ability to influence without authority and persuade engineers to adopt new practices.
- Collaborative mindset, preferring to co-own systems.
- Strong user empathy and focus on solving problems for others.
- Strong product instincts, writing skills, and comfort operating autonomously in ambiguous areas.
- Passion for self-driving technologies and frontier AI.