Research, Post-Training Evals

About the job

Responsibilities

• Create internal evaluations and research signals for model capabilities and behaviors relevant to research and post-training.
• Develop usability evaluations to measure model usefulness in real research and product workflows, and collaborate with the data flywheel to improve data and training signals based on evaluation insights.
• Enhance evaluation robustness by addressing grader reliability, ambiguous ground truth, evaluator disagreement, false positives/negatives, and discrepancies between measured and intended behavior.
• Build benchmark auditing methodologies to help researchers understand, trust, and appropriately utilize evaluation signals.
• Develop specialized agentic evaluation environments and user simulators, including supporting harness development and studying generalization across users, harnesses, and environments.
• Develop evaluations for personalized preferences, biases, values, and other nuanced model behaviors in collaboration with the post-training team.

Requirements

• Bachelor's degree or equivalent experience in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding.
• Experience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or other measurement systems.
• Strong written and verbal communication skills, with the ability to collaborate effectively across research and engineering teams.
• Experience with LLMs, post-training, reinforcement learning, or agentic systems.
• Experience with evaluation auditing, human evaluations, LLM-judges, or open-ended task evaluation.
• Experience with agentic evaluation, harnesses, long-horizon tasks, or RL environments.
• Experience evaluating preferences, personalization, biases, values, or other nuanced model behaviors.
• Track record of developing new evaluation methodologies or research signals that meaningfully influenced model development.
• Proficiency in Python and familiarity with at least one deep learning framework (e.g., PyTorch, TensorFlow, or JAX).
• Comfortable with debugging distributed training and writing code that scales.
• Strong research judgment, including clean ablations, honest baselines, and clear technical writing.
• PhD in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding; or, equivalent industry research experience.

Benefits

• Generous health, dental, and vision benefits
• Unlimited PTO
• Paid parental leave
• Relocation support as needed