ML Platform Engineer, Distributed Training
$159k - $219k • Remote • Remote • Full Time or Contract
Posted 1d ago
Job Location
Remote
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full Time or Contract
Categories
AI Infrastructure Engineer
About the job
As a Senior AI Engineer, you will be at the forefront of our Generative AI initiatives, treating AI as a software engineering discipline. You will be responsible for the full lifecycle of our AI features, taking them from initial prototype to robust, scalable production services, while solving for real-world constraints like latency, error handling, and cost optimization. You’ll collaborate with a diverse range of clients to translate business needs into high-performance AI architectures. This role requires a blend of deep technical expertise in LLMs and a disciplined Software Engineering approach to ensure our solutions are robust, ethical, and scalable.
Responsibilities
- Work with the client's infrastructure team on the GKE training platform, focusing on GPU node pools, job queueing, scheduling, quota, and capacity from the training workload's perspective.
- Diagnose and fix platform failures at their root cause.
- Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) for optimal training throughput.
- Run, scale, and debug multi-node distributed training using PyTorch (FSDP, DeepSpeed, Megatron or NeMo-style) and Ray Train.
- Profile jobs to identify and eliminate bottlenecks across compute, memory, input pipelines, and interconnects.
- Design and execute performance benchmarks for the cluster and training stack, measuring throughput, scaling efficiency, and cost per run.
- Build and maintain model evaluation harnesses to quantitatively assess changes to the platform or training recipes.
- Instrument training and the platform with telemetry and monitoring, and ensure end-to-end pipeline tests are running against actual user workloads.
- Proactively identify issues, address gaps, and propose solutions for the training platform, keeping the team informed of status and risks.
- Explain and defend design decisions to client engineers and scientists.
- Adapt effectively to changing client priorities and team structures.
Requirements
- 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training.
- Strong Python software engineering skills, including production-quality code, tests, and code review.
- Experience with GKE and GPU scheduling, including Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or similar), and capacity/quota management.
- Proven experience running and debugging multi-node distributed training jobs with PyTorch and Ray Train.
- Familiarity with RDMA and GPU networking, including GPUDirect/RDMA, NCCL tuning, and multi-node GPU topology.
- Experience designing performance benchmarks and model evaluations to inform decisions.
- Deep understanding of system workings, capable of explaining them without over-reliance on AI coding tools.
- Demonstrated track record of ownership, communication, and accountability in a demanding team environment.