Senior AI Engineer, Quality
$200k - $250k • San Francisco, CA or Remote (USA) • FullTime
Posted 6mo ago
Job Location
San Francisco, CA or Remote (USA)
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Fieldguide is seeking a Senior AI Engineer, Quality to own the evaluation infrastructure for their AI agents, ensuring reliable performance at enterprise scale. This role focuses on making evaluations a core engineering capability by building a unified platform, automated pipelines, and production feedback loops. You will work at the intersection of ML engineering, observability, and quality assurance to meet customer demands for rigorous AI agent standards. This position is for engineers who value in-person collaboration and are passionate about building reliable AI systems.
Responsibilities
- Design and build a unified evaluation platform for agentic systems and audit workflows.
- Develop observability systems to track agent behavior, execution, and failure modes in production.
- Create feedback loops to convert production failures into evaluation cases.
- Own the evaluation infrastructure stack, including integration with LangSmith and LangGraph.
- Translate customer problems into concrete agent behaviors and workflows.
- Integrate and orchestrate LLMs, tools, retrieval systems, and logic into reliable agent experiences.
- Build automated pipelines for rapid model evaluation against critical workflows.
- Design evaluation harnesses for complex agentic systems and workflows.
- Implement comparison frameworks for model effectiveness, consistency, latency, and cost.
- Design guardrails and monitoring systems to prevent quality regressions.
- Utilize AI as a core leverage in design, build, test, and iteration processes.
- Prototype quickly and harden systems for enterprise-grade reliability.
- Build evaluations, feedback mechanisms, and guardrails for continuous agent improvement.
- Curate production traces to create evaluation datasets with SMEs and ML Engineers.
- Design prompts, retrieval pipelines, and agent orchestration systems for scalable performance.
- Define and document evaluation standards, best practices, and processes.
- Advocate for evaluation-driven development and facilitate team adoption of evaluation practices.
- Partner with product and ML engineers to integrate evaluation requirements early in development.
- Take full ownership of large product areas.
Requirements
- Belief that evaluations are foundational to building reliable AI systems.
- Understanding that rapid model evaluation is critical for an AI company.
- Treat LLMs, agents, and automation as fundamental engineering building blocks.
- Make data-driven decisions and focus on measuring key metrics.
- Understand the importance of evaluating real production behavior.
- Possess strong product judgment and the ability to make decisions independently.
- Bias towards building working systems quickly.
- Multiple years of experience shipping production software in complex systems.
- Experience with TypeScript, React, Python, and Postgres.
- Experience building and deploying LLM-powered features serving production traffic.
- Experience implementing evaluation frameworks for model outputs.