Applied AI Researcher, Benchmarking
$150k - $250k • San Francisco • FullTime
Posted 1y ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Distyl AI is seeking creative researchers to push the boundaries of AI utilization in enterprise. This role focuses on defining and measuring progress in AI systems, designing evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact. Researchers will construct benchmarks reflecting real-world complexity and explore new paradigms for evaluating intelligent systems, such as adversarial robustness testing and human-in-the-loop assessment. The goal is to establish rigorous methodologies for quantifying emergent capabilities and drive both internal research priorities and industry-wide standards.
Responsibilities
- Design evaluation frameworks to measure reasoning depth, interaction quality, reliability, and operational impact of AI systems.
- Construct benchmarks that reflect real-world complexity for judging new architectures, techniques, and releases.
- Explore new paradigms for evaluating intelligent systems, including adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment.
- Investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capabilities.
- Develop intelligent systems using existing models rather than training or fine-tuning them.
Requirements
- Experience designing and running evaluations, including building or maintaining benchmarks, test suites, or experimental frameworks.
- Strong statistical and analytical rigor for designing fair, reproducible experiments and extracting signal from noisy empirical results.
- Expertise in compound AI systems, agentic collaboration, and associated techniques (ensembling, ReAct, graph-of-thoughts, etc.).
- Proven track record of research results (publications, public work, etc.).
- Daily use of AI tools like ChatGPT, Cursor, and Perplexity to accelerate workflows.
- Strong programming and data analysis skills to build prototypes and perform experiments.
- Bias towards demonstrating results over theoretical discussion.
Benefits
- Base salary range of $150K – $250K, depending on experience, location, and level.
- Meaningful equity.
- Comprehensive benefits package.
- 100% coverage of medical, dental, and vision insurance for employee and dependents.
- Flexible time off.
- Retirement and financial planning benefits (HSA, FSA, commuter accounts, 401(k), financial coaching).
- Comprehensive wellness benefits (physical fitness, mental well-being, fertility and family-building).
- Complimentary in-office lunches and snacks.
- Access to state-of-the-art AI models and generous usage of modern AI tools.
- Ownership of high-impact projects across top enterprises.