ML Infra Engineer, Modeling
Remote • San Francisco • FullTime
Posted 1mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Physical Intelligence is developing foundation models and learning algorithms to power future physically-actuated devices. In this role, you will be instrumental in scaling and optimizing our training systems and core model code. You will take ownership of critical infrastructure for large-scale training, encompassing GPU/TPU compute management, job orchestration, and the creation of efficient, reusable JAX training pipelines. This position offers a hands-on, high-leverage opportunity at the intersection of machine learning, software engineering, and scalable infrastructure, working closely with researchers and model engineers to translate innovative ideas into production-ready training runs.
Responsibilities
- Own training/inference infrastructure, including scheduling, job management, checkpointing, and metrics/logging.
- Scale JAX-based distributed training across TPU and GPU clusters.
- Optimize performance by profiling and improving memory usage, device utilization, throughput, and distributed synchronization.
- Build abstractions for launching, monitoring, debugging, and reproducing experiments to enable rapid iteration.
- Partner with researchers to translate their needs into infrastructure capabilities and guide best practices for large-scale training.
- Contribute to and evolve JAX model and training code for new architectures, modalities, and evaluation metrics.
Requirements
- Strong software engineering fundamentals and experience building ML training infrastructure or internal platforms.
- Hands-on large-scale training experience in JAX (preferred) or PyTorch.
- Familiarity with distributed training, multi-host setups, data loaders, and evaluation pipelines.
- Experience managing training workloads on cloud platforms like SLURM, Kubernetes, GCP TPU/GKE, or AWS.
- Ability to debug and optimize performance bottlenecks across the training stack.
- Strong cross-functional communication and ownership mindset.