Software Engineer, AI Agent & LLM

About the job

Responsibilities

• Build AI-agent experiences for Otter AI Chat to assist users with reasoning, knowledge retrieval, and task completion.
• Enhance the conversational knowledge engine through improved knowledge extraction, annotation, and indexing.
• Develop robust evaluation datasets, automated grading systems, regression tests, and release gates for AI quality.
• Diagnose and resolve failures across models, prompts, retrieval, tools, data pipelines, backend services, and product workflows.
• Construct shared agent infrastructure for orchestration, tracing, debugging, retries, sandboxed execution, and observability.
• Improve task completion rates, correctness, groundedness, reliability, latency, and cost-effectiveness.
• Translate production traces, customer feedback, and usage signals into measurable product and model enhancements.
• Collaborate with AI, product, infrastructure, data, security, and application teams to deliver end-to-end capabilities.

Requirements

• 3+ years of experience in AI Agent engineering, machine learning engineering, or a related field.
• Strong backend or distributed-systems engineering skills.
• Hands-on experience building and shipping AI-agent or LLM-powered products.
• A strong focus on AI quality and experience evaluating nondeterministic systems.
• Ability to use data, traces, logs, and qualitative examples to identify and resolve complex failures.
• Effectively work across services, technical domains, and organizational boundaries.
• Combine strong product judgment with rigorous engineering and evaluation practices.
• Operate with high agency, strong ownership, and a bias toward action.
• Ability to take ambiguous problems from exploration through production launch and continuous improvement.
• Demonstrated ability to use coding agents effectively while rigorously reviewing and controlling the quality of their output is a plus.