AI Research Engineer - Datadog AI Research (DAIR)
Salary not listed • Paris, France • Full-time
Posted 1y ago
This job has been open for more than 60 days. It may no longer be actively hiring, so check with the company before putting a lot into an application.
About the job
Datadog AI Research (DAIR) is seeking a Research Engineer to partner with Scientists in transforming research concepts into functional systems. This role involves building the necessary data, tooling, and infrastructure to support rapid iteration, reliable evaluation, and a seamless transition from prototype to production. The team focuses on high-impact problems in cloud observability and security, specifically in training multimodal foundation models for understanding distributed systems dynamics and post-training models to operate autonomously within Datadog's domain, starting with SRE incident response.
Responsibilities
- Build and operate multimodal data pipelines, training and evaluation infrastructure, benchmarks, and internal tooling.
- Implement models, run experiments at scale, and profile for reliability, performance, and cost.
- Build simulation environments and replay infrastructure for agent training and evaluation.
- Orchestrate distributed training and distributed RL with Ray, including scheduling, scaling, and failure recovery.
- Establish rigorous automated benchmarks and regression tests for world model predictions, agent performance, and simulation fidelity.
- Collaborate with Research Scientists, Product, and Engineering to integrate capabilities into Datadog's products and harden prototypes into reliable services.
- Contribute to research publications at top-tier conferences and produce high-quality code, documentation, and open-source artifacts.
Requirements
- Depth in distributed computing, RL Infra, and ML systems for training and inference at scale; experience with Ray, Slurm, or similar frameworks is a plus.
- Proficiency in Python, familiarity with a systems language (e.g., Rust, C++, or Go), and comfort with modern cloud and data infrastructure.
- Practical experience implementing and operating ML training and inference systems (e.g., PyTorch or JAX), including containerization, orchestration, and GPU acceleration.
- Practical experience with large-scale model training and fine-tuning, including frameworks like Megatron-LM, DeepSpeed, SkyRL, VeRL, or TorchTitan, and techniques such as SFT, RLVR, RLHF, and efficient inference (quantization, speculative decoding).
- Ability to clearly explain design and performance trade-offs to both technical and non-technical audiences.
- Experience supporting or contributing to research publications.
- Strong software engineering skills with experience in domains such as observability, SRE, or security (bonus).
- Experience bridging research prototypes and real-world product applications, especially with large foundation models, world models, or RL-trained agents (bonus).
- Hands-on experience with GPU programming and optimization, including CUDA (bonus).
- Experience writing production data pipelines and applications (bonus).
- Experience building simulation or sandbox environments for agent training (bonus).
Benefits
- Competitive global benefits
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Opportunity to collaborate closely with colleagues across the Datadog offices in New York City and Paris
- Opportunity to attend and present at conferences and meetups
- Intra-departmental mentor and buddy program for in-house networking
- An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)