metric validation Jobs

Alpaca is seeking a Senior Data Scientist, AI Evaluation to lead the measurement of our AI models and agents. In this senior individual contributor role, you will transform ambiguous quality questions into reliable ground truth, scoring methods, and evaluation loops that drive company-wide improvements. You will leverage an existing data foundation to enhance quality and accelerate the safe rollout of AI systems, owning the quality bar independently. This role is ideal for someone passionate about the correctness of AI outputs as much as their generation, who can partner with Product, Engineering, Analytics Engineering, and business stakeholders to define success, build robust evaluations, and ensure feedback loops drive iteration. If you have a strong quantitative background, a proven track record of rigorous, measurable work in evaluation, experimentation, or model validation, and desire ownership of a new evaluation practice at a rapidly growing financial services infrastructure company, we encourage you to apply.