Applied Research Engineer, Evaluation
€150k - €220k • Freiburg (Germany) • Full-time
Posted 18d ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Research Engineer
About the job
Black Forest Labs is building the foundational layer for visual intelligence, developing multimodal models trained on images, video, audio, and action. Our team pioneered latent diffusion and Stable Diffusion, making generative AI accessible. We are pushing generative media at scale and developing machines that act in the physical world. This role focuses on the critical challenge of evaluating the performance and quality of frontier image and video models, as public benchmarks have fallen behind the capabilities of current models. You will be instrumental in building and refining the evaluation system that assesses every model checkpoint, finetuning run, and competitor release, providing essential insights for product research and development.
Responsibilities
- Build and maintain the evaluation stack for all model checkpoints, including automated benchmarking and LLM-as-judge pipelines.
- Design evaluations across various aspects such as prompt adherence, editing fidelity, aesthetics, text rendering, consistency, and safety.
- Evaluate the evaluation stack itself, ensuring agreement between human and automated evaluations and improving them when necessary.
- Continuously benchmark FLUX models against frontier models, providing meaningful insights into comparative performance.
- Identify model output shortcomings not captured by aggregate numbers.
- Collaborate with RL and finetuning researchers to define target metrics and verify improvements.
- Design strategies to exploit existing API and MCP signals and identify new signals to capture.
- Maintain the usefulness of existing metrics and anticipate issues like gaming and overfitting for new metrics.
Requirements
- Demonstrated meaningful contributions to the evaluation of generative models.
- Understanding of the limitations of benchmark figures without careful consideration of sample size and judge agreement.
- Advanced degree (MS/PhD) in a relevant field or equivalent practical experience.
- Practical experience evaluating generative models, potentially including diffusion or flow-matching based image or video models.
- Very strong Python and software engineering skills, with experience building reliable pipelines.
- Working knowledge of PyTorch and modern ML infrastructure and practices.
- Strong grasp of statistics and its practical application, including sample sizes, inter-annotator agreement, and confidence levels.
- Practical experience building and operating LLM-as-judge systems and understanding their limitations.
- Working knowledge of modern benchmarking practices, standard datasets, and benchmarks.
- Inclination to examine actual model outputs beyond aggregate scores.