Applied AI Software Engineer, GTM Growth Engineering
San Francisco • FullTime
Posted 1mo ago
About the job
We are seeking an Applied AI Engineer to develop production systems that enhance AI-powered go-to-market workflows. This role involves connecting agent behavior, customer and operator feedback, evaluation, and business outcomes to create more effective, reliable, and responsive systems. You will take end-to-end ownership of the agent improvement loop, from understanding production behavior and identifying failure modes to improving system decisions and validating impact. This is a deeply technical, cross-functional position where you will partner with Engineering, Product, Data Science, Sales, and B2B Marketing to translate real-world signals into safer, more effective agent behavior and measurable business improvements.
Responsibilities
- Own the production improvement loop for agent behavior, feedback, evaluation, experimentation, and business outcomes.
- Instrument agent workflows to understand model interactions, tool use, decisions, failures, human edits, and downstream outcomes.
- Define quality standards, evaluation datasets, regression coverage, and production monitoring for GTM workflows.
- Investigate agent underperformance across various aspects like context, knowledge, instructions, tools, routing, guardrails, or workflow design.
- Design and implement targeted behavior improvements, including changes to prompting, context construction, decision logic, tool use, and human-review paths.
- Build backend services, APIs, data models, and feedback pipelines for observable, steerable, and reproducible agent behavior.
- Run controlled experiments, production replays, or staged rollouts to measure the impact of changes on quality and business results.
- Partner with cross-functional teams to prioritize high-value problems and define success metrics.
- Ship improvements with appropriate safeguards for privacy, security, reliability, human oversight, and safe operational rollout.
Requirements
- 4+ years of experience in software, backend, applied AI, or product engineering, building reliable production systems.
- Experience building AI agents, LLM-powered applications, or model-driven workflows on production traffic.
- Experience diagnosing and improving agent behavior using production traces, user feedback, evaluation, experimentation, or systems design.
- Practical experience with evaluation design, regression testing, human or model grading, online quality signals, or controlled experiments.
- Strong backend engineering skills in Python, APIs, data pipelines, stateful workflows, and production services.
- Strong product judgment and ability to link technical changes to customer experience and business outcomes.
- Comfort working with model behavior, context, knowledge, tools, workflow state, and human-in-the-loop decisions.
- Ability to collaborate effectively with technical and non-technical partners.
- Pragmatic mindset for scoping ambiguous problems, shipping improvements, and building durable systems.