Staff Engineer, Machine Learning Systems & Reliability - Moveworks
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
About the job
We are seeking a Staff Engineer to build and maintain the production systems that power our AI-enabled product capabilities. This role bridges ML systems, platform engineering, and site reliability engineering, focusing on moving machine-learning models and agentic workflows from prototypes to secure, observable, and continuously deployable production systems. You will partner with various teams to create a streamlined path from experimentation to production and ensure the ongoing performance and evolution of these systems.
Responsibilities
- Design and build the production pipeline for the complete ML lifecycle, including data preparation, training, experiment tracking, evaluation, model management, serving, monitoring, and retraining.
- Develop continuous-delivery workflows for models, prompts, agent workflows, and supporting services, incorporating automated quality, safety, and performance checks.
- Implement safe rollout strategies such as shadow traffic, canaries, progressive delivery, and automated rollback mechanisms.
- Transform self-learning approaches into controlled production feedback loops with systems for outcome collection, feedback validation, lineage maintenance, and model refresh triggering.
- Define and operate SLIs, SLOs, alerts, and error budgets across infrastructure, data pipelines, inference services, model quality, and product behavior.
- Integrate model analytics and product telemetry with operational signals to diagnose issues across infrastructure, data, model behavior, or product.
- Enhance the scalability, availability, latency, and cost efficiency of distributed training, inference, and data-processing workloads, including capacity planning and resource optimization.
- Participate in the full service lifecycle, including architecture reviews, deployment, on-call rotations, incident response, postmortems, and remediation.
- Build self-service platforms and automation to reduce operational toil and accelerate the deployment of ML engineers and data scientists.
- Apply LLMs or agentic automation to improve evaluation, troubleshooting, and operational workflows.
- Establish practical standards for cloud infrastructure, Kubernetes, infrastructure as code, observability, security, and compliance.
- Provide technical leadership, mentor engineers, and influence architecture across ML, data, product, and platform teams.
Requirements
- 7+ years of experience in software engineering, platform engineering, SRE, production engineering, or ML infrastructure, with a track record of Staff-level technical ownership.
- Strong software engineering skills in Python and at least one production systems language (e.g., Go, Java, C++, Rust).
- Experience designing, operating, and troubleshooting distributed production systems, including failure analysis, capacity planning, and performance optimization.
- Hands-on experience with cloud infrastructure, containers and Kubernetes, infrastructure as code, CI/CD, and modern observability tools.
- Practical understanding of the ML lifecycle (training, evaluation, deployment, serving, monitoring, versioning, retraining) and ability to collaborate with ML engineers or researchers.
- Experience differentiating service-health problems from data-quality or model-quality issues.
- Familiarity with SRE practices like SLIs/SLOs, error budgets, sustainable on-call, incident management, and blameless postmortems.
- Strong automation and internal-customer mindset, focused on building reliable and user-friendly platforms.
- Excellent technical judgment and communication skills, especially in ambiguous situations and during production incidents.