Senior ML Systems Engineer, Frameworks & Tooling
Remote • London • FullTime
Posted 8mo ago
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Machine Learning Engineer
About the job
Cohere is seeking a Senior ML Systems Engineer to join their team and build, maintain, and evolve the training framework that powers their frontier-scale language models. This role is ideal for someone passionate about large-scale training, distributed systems, and HPC infrastructure, offering the opportunity to design and maintain core components for fast, reliable, and scalable model training. You will also build tooling to connect research ideas to thousands of GPUs, working across the full stack of ML systems with significant autonomy and impact.
Responsibilities
- Build and own the training framework for large-scale LLM training.
- Design distributed training abstractions including data/tensor/pipeline parallelism, FSDP/ZeRO strategies, memory management, and checkpointing.
- Improve training throughput and stability on multi-node clusters.
- Develop and maintain tooling for monitoring, logging, debugging, and developer ergonomics.
- Collaborate with infrastructure teams to ensure cluster, container environments, and hardware configurations support high-performance training.
- Investigate and resolve performance bottlenecks across the ML systems stack.
- Build robust systems for reproducible, debuggable, large-scale runs.
Requirements
- Strong engineering experience in large-scale distributed training or HPC systems.
- Deep familiarity with JAX internals, distributed training libraries, or custom kernels/fused ops.
- Experience with multi-node cluster orchestration (Slurm, Ray, Kubernetes, or similar).
- Comfort debugging performance issues across CUDA/NCCL, networking, IO, and data pipelines.
- Experience working with containerized environments (Docker, Singularity/Apptainer).
- Track record of building tools that increase developer velocity for ML teams.
- Excellent judgment around trade-offs (performance vs complexity, research velocity vs maintainability).
- Strong collaboration skills with infra, research, and deployment teams.
Benefits
- Weekly lunch stipend of $75/£75 or equivalent.
- Full health and dental benefits, including a separate budget for mental health.
- RRSP matching, 401K, Pension Scheme.
- 100% Parental Leave top-up for up to 6 months.
- Annual enrichment benefits (Arts & culture, fitness/wellness, quality time, workspace improvement).
- Education & learning stipend for conferences, courses, and coaching.
- 6 weeks of paid vacation (30 working days).
- Budget for traveling to other offices and an annual company offsite.
- Co-working benefit for remote employees not near an office.
- $500 home office stipend.