Member of Technical Staff - Distributed Training Engineer
Remote • San Francisco • FullTime
Posted 1y ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Liquid AI, spun out of MIT CSAIL, is developing general-purpose AI systems designed for efficient execution across various deployment targets, from data center accelerators to on-device hardware. The company focuses on ensuring low latency, minimal memory usage, privacy, and reliability, partnering with enterprises in consumer electronics, automotive, life sciences, and financial services. The Training Infrastructure team is building the distributed systems that power next-generation Liquid Foundation Models, requiring the design, implementation, and optimization of large-scale training infrastructure. This role is focused on runtime, performance, and reliability within training systems, offering a high-ownership opportunity to build critical systems from the ground up within a small team with fast feedback loops.
Responsibilities
- Design and build core systems for fast and reliable large training runs
- Build scalable distributed training infrastructure for GPU clusters
- Implement and tune parallelism/sharding strategies for evolving architectures
- Optimize distributed efficiency, including topology-aware collectives, communication/computation overlap, and straggler mitigation
- Build data loading systems to eliminate I/O bottlenecks for multimodal datasets
- Develop checkpointing mechanisms that balance memory constraints with recovery needs
- Create monitoring, profiling, and debugging tools for training stability and performance
Requirements
- Hands-on experience building distributed training infrastructure (e.g., PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, Megatron-LM TP/PP)
- Experience diagnosing performance bottlenecks and failure modes (profiling, NCCL/collectives issues, hangs, OOMs, stragglers)
- Understanding of hardware accelerators and networking topologies
- Experience optimizing data pipelines for ML workloads
- Experience with MoE (Mixture of Experts) training
- Experience with large-scale distributed training (100+ GPUs)
- Open-source contributions to training infrastructure projects
Benefits
- Competitive base salary
- Equity in a unicorn-stage company
- 100% coverage of medical, dental, and vision premiums for employees and dependents
- 401(k) matching up to 4% of base pay
- Unlimited PTO
- Company-wide Refill Days