Senior Data Scientist AI Evaluation
Remote • Remote - Americas
Posted 10d ago
Job Location
Remote - Americas
Tech Stack
Remote Work Policy
Fully remote
Categories
Applied AI Engineer
About the job
Alpaca is seeking a Senior Data Scientist, AI Evaluation to lead the measurement of our AI models and agents. In this senior individual contributor role, you will transform ambiguous quality questions into reliable ground truth, scoring methods, and evaluation loops that drive company-wide improvements. You will leverage an existing data foundation to enhance quality and accelerate the safe rollout of AI systems, owning the quality bar independently. This role is ideal for someone passionate about the correctness of AI outputs as much as their generation, who can partner with Product, Engineering, Analytics Engineering, and business stakeholders to define success, build robust evaluations, and ensure feedback loops drive iteration. If you have a strong quantitative background, a proven track record of rigorous, measurable work in evaluation, experimentation, or model validation, and desire ownership of a new evaluation practice at a rapidly growing financial services infrastructure company, we encourage you to apply.
Responsibilities
- Design AI evaluations, defining ground truth, metrics, and scoring methods for models and agents.
- Build repeatable evaluation loops to track quality over time and detect regressions before release.
- Partner with engineering and analytics engineering to operationalize evaluation harnesses.
- Translate evaluation results into actionable recommendations for system improvements.
- Establish evaluation guidelines, documentation, and review practices to set quality standards.
- Foster evaluation best practices and promote a culture of measurable AI quality across the team.
Requirements
- Proven track record of quantitative measurement rigor, such as LLM/model evaluation, metric validation, or experimentation.
- Strong statistical and ML foundation, treating evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validating automated graders against human ground truth.
- Proficiency in Python and SQL, with experience evaluating models in production environments.
- Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
- Excellent communication and cross-functional collaboration skills to align technical teams and leadership.
- Strong problem-solving ability in fast-paced, greenfield environments.
- 6-10 years in quantitative data science or ML, with focused experience in measurement or evaluation.
- A quantitative degree is a plus; equivalent industry experience is equally welcome.
Benefits
- Competitive Salary
- Stock Options
- Health Benefits
- New Hire Home-Office Setup: One-time USD $500
- Monthly Stipend: USD $150 per month via a Brex Card