Staff Software Engineer, AI Runtime
$190k - $265k • Remote • Mountain View, California; San Francisco, California
Posted 3d ago
Job Location
Mountain View, California; San Francisco, California
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
Databricks is seeking a Staff Software Engineer for its AI Runtime (AIR) team. AIR is a managed platform designed for large-scale GPU training and fine-tuning of state-of-the-art AI models, providing customers with on-demand access to accelerators and a serverless experience. As part of this role, you will be instrumental in building and scaling the systems that ensure fast, reliable, and effortless large-scale training. You will influence the architecture and development of the managed GPU training stack, focusing on areas like scheduling, distributed training performance, fault tolerance, and the overall developer experience for operating jobs at scale. This position also involves defining the long-term technical vision for AIR, mentoring senior engineers, and collaborating with product, research, and platform teams to expand the impact of custom training at Databricks.
Responsibilities
- Drive the architecture and evolution of AIR's managed GPU training platform for scalable, high-throughput, and resilient training.
- Solve complex problems in large-scale training, including multi-node orchestration, distributed parallelism, GPU scheduling, data loading, and checkpointing for long-running jobs.
- Improve GPU efficiency and training performance to increase utilization and reduce costs.
- Build resilience and observability foundations for multi-node jobs to detect and recover from failures.
- Collaborate with product, research, and platform teams to enhance APIs, CLI, and developer experience for training jobs.
- Lead end-to-end engineering efforts from design to production rollout, ensuring performance, correctness, and reliability.
- Contribute directly to core AIR systems and support new accelerators and regions.
- Champion engineering excellence, mentor engineers, and shape the long-term technical direction in AI training infrastructure.
Requirements
- 10+ years of experience in building and operating large-scale distributed systems, with depth in GPU training infrastructure, HPC, or ML systems.
- Hands-on experience with distributed training frameworks (e.g., PyTorch, FSDP, DeepSpeed, Megatron) and parallelism strategies.
- Strong understanding of training resilience patterns, including checkpointing, failure detection, and automatic recovery.
- Solid grasp of GPU performance fundamentals, including accelerator architecture, high-speed interconnects, collective communication, and performance bottlenecks.
- Experience building and operating managed, multi-tenant cloud platform products with clear SLAs/SLOs.
- Strong foundation in algorithms, data structures, and system design for performance-sensitive distributed systems.
- Proven ability to deliver technically complex, high-impact initiatives.
- Strong communication and collaboration skills across diverse teams.
- Strategic, product-oriented mindset with a passion for mentoring and technical excellence.
- BS in Computer Science or related field (MS or PhD preferred).