Lead Engineer, AI Platform
$170k - $170k • Remote • Anywhere • Full-time
Posted 1d ago
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
Circle is seeking a Lead Engineer for its AI Quality engineering team. This team is responsible for building the infrastructure to measure, diagnose, and improve the quality of Circle's AI-powered features, focusing on production AI systems rather than ML research or model training. The role involves building evaluation frameworks, observability tooling, and diagnostic infrastructure to ensure AI Agents are working effectively, efficiently, and at scale. This is a hands-on player-coach position where you will also lead and manage the AI Quality engineering team, contributing to an ambitious roadmap.
Responsibilities
- Build and own evaluation infrastructure, including CI/CD pipelines, scorers, and datasets for AI agents.
- Diagnose quality breakdowns across the agent pipeline, from plan creation to execution and tool usage.
- Grow datasets for evaluation through annotation workflows, AI-generated data, and simulated conversations.
- Run structured experiments on prompts, models, and agent harnesses to evaluate new models and optimize for cost and latency.
- Lead and grow the AI Quality engineering team, setting technical direction and managing priorities while remaining hands-on.
- Partner with AI Core engineering to ensure confidence in quality improvements.
- Design and implement CI/CD pipelines for AI evaluation.
- Develop scorers and datasets to assess AI agent performance.
- Trace failures across complex agent pipelines.
- Establish annotation workflows for dataset creation.
- Evaluate open-source models against production baselines.
- Implement model swaps, caching, and routing strategies for cost and latency optimization.
Requirements
- 7+ years of experience building and shipping production software, including LLM-powered agents that take real actions.
- Experience with complex, tool-using systems involving multiple tools, planning, or orchestration.
- Proficiency in Ruby on Rails or Python, or ability to learn quickly.
- Experience building evaluation or observability infrastructure for ML/AI systems.
- Experience with evaluation frameworks like Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
- Ability to learn fast, experiment aggressively, and thrive in a dynamic environment.
- Comfortable in a fast-paced environment with ambiguity and learning new technologies.
- Proficient in English (CEFR Level C2 / ILR Level 5).
Benefits
- U.S.-benchmarked compensation globally
- Equity in the company with ongoing refresh grants
- 35 days of paid time off annually
- Company retreats twice a year
- Flexibility and autonomy
- Benefits package supporting health, wellbeing, and professional growth