Senior Software Engineer, AI Runtime
$160k - $225k • Remote • Mountain View, California; San Francisco, California
Posted 5d ago
Job Location
Mountain View, California; San Francisco, California
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
Databricks is seeking a Senior Software Engineer for its AI Runtime (AIR) team. AIR is a managed platform designed for large-scale GPU training and fine-tuning of AI models, providing customers with on-demand access to accelerators and a serverless experience that simplifies the complexities of provisioning, scheduling, and orchestrating multi-node jobs. As a Senior Software Engineer on AIR, you will be instrumental in building and scaling the systems that ensure large-scale training is fast, reliable, and effortless. You will lead the architecture and evolution of the managed GPU training stack, focusing on areas such as scheduling, distributed training performance, fault tolerance, and the developer experience for large-scale job operations. Beyond hands-on system development, you will contribute to the technical vision for AIR, mentor engineers, and collaborate with product, research, and platform teams to expand the impact of custom AI training at Databricks.
Responsibilities
- Drive the architecture and evolution of AIR's managed GPU training platform for scalable, high-throughput, and resilient training.
- Solve complex problems in large-scale training, including multi-node orchestration, distributed parallelism, GPU scheduling, data loading, and checkpointing.
- Improve GPU efficiency and training performance to increase utilization and reduce costs.
- Build resilience and observability foundations for multi-node jobs to detect and recover from failures.
- Collaborate with product, research, and platform teams to enhance the developer experience for launching and monitoring training jobs.
- Lead end-to-end engineering efforts from design to production rollout, ensuring high standards for performance, correctness, and reliability.
- Contribute directly to core AIR systems and support new accelerators and regions.
- Promote engineering excellence, mentor other engineers, and influence Databricks' technical direction in AI training infrastructure.
Requirements
- 5+ years of experience building and operating large-scale distributed systems, with experience in GPU training infrastructure, HPC, or ML systems.
- Experience with distributed training frameworks (e.g., PyTorch, FSDP, DeepSpeed, Megatron) and parallelism strategies.
- Strong understanding of training resilience patterns, including checkpointing, failure detection, and recovery for long-running jobs.
- Solid grasp of GPU performance fundamentals, including accelerator architecture, high-speed interconnects, collective communication, and bottlenecks.
- Experience building and operating managed, multi-tenant cloud platform products with clear SLAs/SLOs.
- Strong foundation in algorithms, data structures, and system design for performance-sensitive distributed systems.
- Proven ability to deliver technically complex, high-impact initiatives.
- Strong communication and collaboration skills across diverse teams.
- Customer-focused mindset and passion for mentoring engineers.
- BS in Computer Science or related field (MS or PhD preferred).