AI Engineer

About the job

Responsibilities

• Fine-tune models for long-running agents, including those that diagnose test failures based on infrastructure, code changes, or authorship.
• Build and curate training and evaluation datasets from production workflows, identifying improvement opportunities through failure patterns.
• Develop LLM evaluations and observability to link offline results with production behavior and validate improvements.
• Tune prompts, context, and model behavior to enhance task success, reliability, latency, token usage, and cost.
• Design repeatable experiments and contribute to publishing benchmarks showcasing core agent performance.
• Own technical projects from inception to delivery, making practical trade-offs with the engineering team.

Requirements

• Relevant software engineering experience (typically 3+ years) with hands-on contributions to production AI capabilities.
• Hands-on experience with supervised fine-tuning, including data preparation and model behavior evaluation.
• Experience with prompt and context tuning, evaluation datasets, and LLM observability for failure diagnosis and improvement measurement.
• Strong engineering fundamentals, efficient coding practices, and self-awareness of strengths and growth areas.
• Experience using AI agents for software development, including task breakdown, progress tracking, result checking, and agent redirection.
• Comfort with autonomy and ambiguity, with the ability to translate unclear problems into actionable plans and manage work independently.

Benefits

• Medical, dental, and vision insurance.
• 401(k).
• Unlimited PTO and sick days.
• Company-observed US holidays.
• Daily lunch.
• Fully stocked kitchen.
• Commuter benefits.
• Team offsites and events.