New: upload your resume and get matched to AI jobs. Free to start →

Machine Learning Infrastructure Engineer

$200k - $400k • San Francisco • Full-time

Posted 1d ago

Job Location

San Francisco

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Infrastructure Engineer

About the job

About the Role

Join an AI team building large physics foundation models to predict real-world systems and identify effective actions. You will develop the distributed infrastructure that supports model training and inference, with weather data as a key application area.

What You'll Do

  • Design, deploy, and maintain distributed machine learning training and inference clusters.
  • Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the ML lifecycle.
  • Evaluate training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
  • Profile and debug low-level GPU operations to improve performance.
  • Track relevant research and apply promising ideas to infrastructure and workflows.

What We're Looking For

  • Two to ten years of experience building large-scale infrastructure for pretrained or foundation models, including compute clusters, data infrastructure, inference, and distributed training.
  • Hands-on experience optimizing large-scale training, data, or inference workloads, and using distributed training frameworks such as FSDP or DeepSpeed.
  • Experience with low-level GPU performance analysis and debugging, including CUDA and JAX.
  • Familiarity with Python, C++, Linux, NVIDIA GPUs, cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.
  • Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications.
  • Comfort working across the ML lifecycle in a fast-paced environment, including analyzing experiments and ablation studies.

Compensation & Benefits

Base salary is $200,000 to $400,000 annually. Visa sponsorship is not available.

Location

On-site in San Francisco, California. Relocation support is provided.

About True Talent Solutions Partners