Senior AI R&D Engineer - Performance - Pre-training (f/m/d)
Heidelberg • FullTime
Posted 2d ago
Job Location
Heidelberg
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Aleph Alpha is seeking a Senior Performance Engineer to join their pre-training efficiency team in Heidelberg. This role focuses on optimizing the systems required to train foundation models at scale, aiming to maximize hardware utilization and training throughput on large-scale GPU clusters. You will work at the intersection of deep learning frameworks, distributed systems, and GPU microarchitecture to eliminate bottlenecks from the Python layer down to the GPU kernel. The team fosters a culture of ownership, autonomy, and empowerment, with a flat organizational structure that enables quick decision-making and open communication.
Responsibilities
- Profile training loops using PyTorch Profiler, Nsight Systems, and Nsight Compute to identify and resolve system- and kernel-level bottlenecks, maximizing model throughput.
- Configure and tune composite parallelism strategies (e.g., TP, DP, HSDP/FSDP, EP) for optimal load balance, minimal critical-path bottlenecks, and effective communication-to-computation trade-offs in large-scale LLM training.
- Collaborate with AI Researchers to define hardware-efficient model architectures that maintain convergence.
Requirements
- Proficiency in Python and the PyTorch library.
- Strong engineering background in parallel and/or distributed systems with a proven track record of excellence.
- Hands-on experience with modern machine learning techniques, particularly large language models and their lifecycle.
- Deep understanding of the CUDA programming model.
- Experience in distributed programming with APIs like NCCL or MPI.
- Experience analyzing profiling traces with tools such as PyTorch Profiler and Nvidia Nsight.
- Contributions to modern distributed training frameworks (e.g., TorchTitan, Megatron-LM, DeepSpeed).
- Familiarity with low-precision training formats (MXFP4, MXFP8) and their impact on numerical stability and throughput.
- Deep understanding of NCCL communication primitives, NVSHMEM, or CUDA IPC and their performance.
- Proven track record of implementing and optimizing modern transformer-based model training.
- Proven track record working on the NVIDIA Blackwell architecture.
Benefits
- 30 days of paid vacation
- Access to fitness & wellness offerings via Wellhub
- Mental health support through nilo.health
- Substantially subsidized company pension plan
- Subsidized Germany-wide transportation ticket
- Budget for additional technical equipment
- Flexible working hours
- Hybrid working model
- JobRad® Bike Lease