Software Engineer, Applied AI
Remote • New York • FullTime
Posted 1mo ago
About the job
Clay is seeking a Software Engineer to join their Applied AI team, focusing on building and refining AI agents that go beyond simple text generation to perform complex, multi-step tasks for users. This role involves working closely with product and research teams to ensure these agents are not only capable but also reliable, steerable, and trustworthy for production use. The core challenge is bridging the gap between demo-worthy agents and those that can operate dependably unattended, translating model improvements into tangible benefits like task completion, reliability, and time savings for end-users.
Responsibilities
- Design and iterate on agent behavior for real go-to-market workflows, handling ambiguous data and making judgment calls.
- Map manual, multi-step workflows and transform them into agent-driven flows that match or exceed human performance.
- Build and run evaluations to measure task completion accuracy and identify regressions.
- Analyze production failures and systematically improve agent robustness.
- Collaborate with product teams to move agent flows from prototype to general availability.
- Build the core agent harness, including memory systems, tool infrastructure, and retrieval architecture.
- Improve agent performance through advanced prompting strategies, tool-use design, and context construction.
- Design and implement guardrails and safety checks for predictable agent behavior in production.
- Develop a cross-surface evaluation framework for consistent quality measurement across teams.
- Build feedback loops to leverage production logs and usage data for prompt and tool improvements.
- Support teams developing custom agent variants for specific use cases.
Requirements
- Experience building or shipping production systems with LLMs or agents, including multi-step orchestration, prompting, tool-use design, retrieval, structured extraction, or fine-tuning.
- Strong backend fundamentals in APIs, databases, and distributed systems.
- Experience with model or agent evaluation, including designing evals, measuring regressions, and quantifying quality.
- A systems-and-outcomes mindset focused on user product functionality.
- Comfort debugging complex real-world failures and a bias for rapid iteration.
- Experience with agent frameworks, tool-calling systems, or retrieval architectures (e.g., vector search, RAG).
- Experience building or maintaining eval/benchmark infrastructure for LLM systems or running fine-tuning in production.
- Familiarity with GTM, sales, or marketing workflows.
- Familiarity with Clay's technology stack (React, TypeScript, Python, AWS services, Terraform, Datadog).
Benefits
- Work with world-class coaches specializing in creativity, management, and more.
- Access to a community of 11,000+ customers, 150+ integration partners, 125+ agencies, 50+ Clay clubs, and 30k members on Slack.
- Opportunity to work on cutting-edge AI agent technology.
- Collaborative and unique company culture.