Distributed Training Jobs

Groq is building high-performance AI infrastructure designed to make inference fast, predictable, and scalable. Our infrastructure teams design and operate the systems that connect compute, data, and services at significant scale. We are looking for a Sr./Staff Network Engineer, AI/HPC to help architect, design, validate, and evolve the network infrastructure supporting Groq's rapidly growing AI compute environments. As a Sr./Staff Network Design Engineer, you will own network architecture and design across high-performance data center and AI infrastructure environments. You will translate capacity, performance, reliability, and business requirements into scalable network designs that can be deployed and operated consistently. This role requires deep expertise in data center networking, routing protocols, network architecture, automation, and large-scale infrastructure. You will work closely with network operations, data center engineering, compute, platform, security, and infrastructure teams to take designs from concept through validation and production deployment.

$270k - $366k

NVIDIA is seeking versatile software engineers for its XLA team to build high-performance, production-grade software at the core of next-generation AI systems. This role involves developing compiler optimization algorithms for deep learning workloads, optimizing inference and training performance for the JAX framework and the OpenXLA compiler on NVIDIA GPUs at scale. You will collaborate with deep learning framework and hardware architecture teams to accelerate future AI software.

$152k - $288k

NVIDIA is at the forefront of the AI revolution, exploring the next frontier of Physical AI, which involves machines that can perceive, reason about, and act within the three-dimensional world. The NVIDIA Cosmos platform provides open foundation models for Physical AI, designed to interpret images, video, and text to create a structured understanding of physical scenes, including motion, object interactions, geometry, and physical context. These models serve as the reasoning layer for robots, autonomous vehicles, and smart infrastructure. The Cosmos Engineering team is responsible for developing the core capabilities behind these models and is seeking a Senior Deep Learning Engineer to lead the data engine and end-to-end training verification for 3D spatial reasoning and perception. This role involves ensuring models accurately learn and represent 3D spatial concepts, working closely with research scientists to translate hypotheses into measurable capabilities in shipped models, and focusing on the critical role of data and evaluations in achieving frontier model quality.

LanceDB is seeking an AI Research Engineer to collaborate with their research team on open-ended research projects. The role focuses on end-to-end training flows across various AI domains, demonstrating how LanceDB can accelerate research. This is a unique opportunity to pursue research interests as an engineer, significantly impacting product development and raising awareness of LanceDB's capabilities.

$150k - $255k

OpenAI

We are seeking an AI Systems Engineer to join our pAGI Infra team, which builds and operates the systems essential for large-scale model training and evaluation. This role involves owning projects from identifying bottlenecks and designing solutions to deployment and operation, focusing on distributed systems engineering, performance optimization, and collaboration with researchers. You will directly contribute to improving the speed and reliability of research by potentially building shared grading services, optimizing resource allocation, or integrating new training stacks into production.

$266k - $500k

OpenAI

We’re hiring a Research Engineer to help OpenAI models solve chip-design problems through reinforcement learning, tool use, and evaluation. You’ll own experiments from the initial idea through implementation and analysis, building environments and evaluations, running training, investigating failures, and using the results to decide what to try next. You’ll also build the software needed to make those experiments reliable and reproducible. We value strong coding fundamentals, careful experimental judgment, and the ability to make progress independently. Prior chip-design experience is helpful, but you can learn the domain alongside the team’s hardware specialists.

$380k - $500k

We are seeking a Staff Engineer to build and maintain the production systems that power our AI-enabled product capabilities. This role bridges ML systems, platform engineering, and site reliability engineering, focusing on moving machine-learning models and agentic workflows from prototypes to secure, observable, and continuously deployable production systems. You will partner with various teams to create a streamlined path from experimentation to production and ensure the ongoing performance and evolution of these systems.

NVIDIA is seeking a Deep Learning Software Infrastructure Engineer to advance its Autonomous Vehicles project. In this role, you will be responsible for building and scaling the training libraries and infrastructure necessary for end-to-end autonomous driving models. By enabling training on thousands of GPUs and massive datasets, you will significantly accelerate iteration speed and enhance safety, working collaboratively with research and platform teams across NVIDIA.

$224k - $431k

Moveworks is seeking a Machine Learning Engineer to develop cutting-edge ML infrastructure for building and serving LLMs. This role is pivotal in constructing, optimizing, and scaling end-to-end machine learning systems, encompassing distributed training and inference pipelines for large language models, model evaluation and monitoring frameworks, and LLM latency optimization. The frameworks developed will serve as a robust foundation for hundreds of ML and NLP models in production, impacting millions of enterprise employees. You will address challenges in service scalability and core algorithm optimization, working closely with machine learning, data infrastructure, and core skill teams. Your contributions will significantly influence customer AI experiences and the long-term scalability of our core AI product.

Lightning AI is seeking an AI Platform Support Engineer to join their US Customer Experience team. This role is designed for individuals who are technical partners to ML teams, helping them diagnose and resolve complex issues related to large-scale training and inference workloads across cloud infrastructure, Kubernetes, and GPU platforms. You will be instrumental in supporting engineers as they train models, deploy inference systems, and scale GPU workloads in production, tackling problems ranging from Kubernetes scheduling and GPU orchestration to distributed PyTorch failures, inference latency, networking, and storage performance. This position offers exposure to diverse AI workloads and the opportunity to shape the infrastructure powering future ML applications.

$115k - $140k

NVIDIA is seeking an NCX Senior Engineer to join our DSX team, collaborating closely with strategic customers to implement and enhance groundbreaking AI workloads. You will deliver hands-on technical assistance for advanced AI deployments, intricate distributed systems, and ensure customers realize efficient performance from NVIDIA's AI platform across varied environments. We partner with the world's most innovative AI companies to address their most challenging technical problems. In this role, you will develop innovative solutions that advance AI infrastructure capabilities and directly influence customer success with breakthrough AI initiatives.

$184k - $357k

This internship program offers a full-time, in-person opportunity in Berlin for 12 to 24 weeks. Interns will focus on advancing search quality through innovative models, data, and tools. The role involves training and optimizing large-scale deep learning models, with a particular emphasis on retrieval and ranking models. Additionally, interns will conduct research in representation learning, covering areas like contrastive learning, multilingual, evaluation, and multimodal modeling for search and retrieval. A key aspect of the role includes building and optimizing RAG pipelines for enhanced grounding and answer generation.

Physical Intelligence is developing foundation models and learning algorithms to power future physically-actuated devices. In this role, you will be instrumental in scaling and optimizing our training systems and core model code. You will take ownership of critical infrastructure for large-scale training, encompassing GPU/TPU compute management, job orchestration, and the creation of efficient, reusable JAX training pipelines. This position offers a hands-on, high-leverage opportunity at the intersection of machine learning, software engineering, and scalable infrastructure, working closely with researchers and model engineers to translate innovative ideas into production-ready training runs.

Thinking Machines is seeking a Site Reliability Engineer (SRE) to ensure the speed, reliability, and ease of iteration for our post-training and reinforcement learning (RL) systems. This role is akin to production engineering or site reliability, focused on model training infrastructure. You will be responsible for the health of training runs, clusters, and pipelines, working closely with research teams to debug issues in real-time, strengthen infrastructure, and develop tools and automation that allow researchers to focus on scientific discovery rather than job management. This position offers significant ownership, requiring you to be the point person for critical issues during active model runs and to implement solutions preventing future occurrences.

$300k - $350k

NVIDIA is seeking outstanding senior engineers to work on the CUDA driver, a key component of accelerated GPU computing. You will join a versatile software engineering team that delivers innovative software features to unlock the full potential and performance of NVIDIA hardware across diverse workloads like deep learning, scientific research, autonomous vehicles, gaming, and virtual reality. This multi-functional role requires collaboration with hardware architects, deep learning specialists, and both internal and external partners to advance the CUDA architecture. With the opportunity to collaborate with teams across the whole NVIDIA computing stack, you will help design software solutions across kernel mode components, compilers, and networking software. Your system-level expertise and creativity in solving complex problems will help invent the future of CUDA and NVIDIA’s compute technologies! Does crafting solutions to influence the evolution of GPU-accelerated computing sound exciting? If so, we invite you to join us and help engineer the next wave of innovation at NVIDIA!

$152k - $288k