Agent Reliability Engineer, GTM Engineering
$150k - $190k • San Francisco, CA • FullTime
Posted 7d ago
About the job
LangChain is seeking an Agent Reliability Engineer to own the health, cost, performance, and business impact of the GTM Agent. You will build feedback loops to ensure continuous improvement and operate the agent on LangSmith, creating a reference story for enterprise customers. This role involves working across a modern tech stack to monitor production health, triage issues, and track key metrics. You will also be responsible for building business metrics to demonstrate the agent's value and developing internal monitoring and alerting systems.
Responsibilities
- Monitor production health of AI agents, identifying and addressing errors, slow runs, high costs, and silent failures.
- Triage and resolve incoming issues from various channels, escalating to the appropriate owners when necessary.
- Run and analyze weekly evaluation suites, investigate failures, and create regression tests for production bugs.
- Track and analyze cost and latency metrics for models and graphs, recommending optimizations.
- Monitor usage and adoption of agent features, and prepare weekly health reports.
- Develop business metrics to quantify the agent's impact on key performance indicators like reply rates, meetings booked, and ROI.
- Build internal monitoring and alerting systems on LangSmith and document best practices for agent operation.
Requirements
- Strong proficiency in production Python and SQL, with experience working with traces, logs, and warehouse tables.
- Experience running LLM applications, including tracing, evaluations, and managing prompt and cache mechanics.
- Solid understanding of SRE or production operations principles, including percentiles and SLOs.
- Ability to critically assess metrics and understand their underlying data.
- Clear and concise writing skills, with an interest in sharing learnings.
- High level of initiative and ability to identify and build missing components.
Benefits
- Medical, dental, and vision coverage
- Flexible vacation
- 401(k) plan
- Meals on in-office days (US)
- Competitive compensation including base salary, variable compensation, and meaningful equity