Member of Technical Staff - Evaluations
San Francisco, CA • FullTime
Posted 7mo ago
Job Location
San Francisco, CA
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Reflection is a research lab dedicated to making intelligence open and accessible for everyone to use, customize, and build on. We are building open models that empower individuals to control their intelligence and shape the future of AI. As a Member of Technical Staff - Evaluations, you will play a critical role in advancing our understanding of model capabilities through rigorous analysis and by building and refining evaluation systems. Your work will create tight feedback loops between data, evaluations, and model behavior, leading to tangible model improvements.
Responsibilities
- Conduct critical comparative analysis to advance understanding of model capabilities.
- Build and refine evaluation systems and processes to create tight feedback loops between data, evals, and model behavior.
- Develop generalizable evaluation frameworks that capture reasoning, alignment, and usefulness.
- Collaborate with pre-training, post-training, and applied teams to translate insights into model improvements.
- Push the boundaries of measurability using synthetic evals, human feedback, and real-world interaction data.
Requirements
- Strong statistical analysis and experimental design skills for rigorous measurement of model improvements.
- Familiarity with LLM evaluation methodologies, including static benchmarks, human preference evals, and/or agentic tasks.
- High agency and ability to thrive in a fast-paced startup environment with a bias for impact.
- Excitement for working in a new frontier lab and defining how progress toward more capable models is measured and accelerated.
- Collaborative, detail-oriented, and motivated by building feedback loops for model improvement.
Benefits
- Top-tier compensation (salary and equity)
- Stock options
- Comprehensive medical, dental, vision, and life insurance
- Annual wellness allowance
- Provided lunch and dinner in the office
- 22 weeks paid parental leave
- Unlimited paid time off (U.S.)
- 30 days paid time off (U.K.)
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations