Software Engineer, Agent Platform
$164k - $306k • Remote • San Francisco, United States • full_time
Posted 29d ago
Job Location
San Francisco, United States
Tech Stack
Remote Work Policy
Fully remote
Employment Type
full_time
Categories
AI Engineer
About the job
Retool is building the future of internal software development, making it safe and efficient to ship custom applications powered by AI. As AI capabilities expand, the challenge shifts from accessing models to ensuring the reliability, evaluability, and trustworthiness of these non-deterministic systems in production. We are seeking AI Engineers to own this critical problem end-to-end. This role is for individuals who are passionate about managing probabilistic systems, measuring quality amidst variance, and driving continuous improvement without introducing regressions.
Responsibilities
- Own the behavior of AI-powered features across multiple product surfaces, ensuring quality, safety, and managing variance and failure modes.
- Design and evolve prompting, retrieval, routing, and tool-use strategies to manage non-determinism effectively.
- Build and maintain evaluation systems to measure model performance using statistical signals and trends.
- Detect, diagnose, and resolve non-deterministic failures like hallucinations, partial correctness, or instruction drift.
- Define and implement guardrails, fallbacks, and degradation paths for systems with unexpected model behavior.
- Partner with product and infrastructure teams to determine when probabilistic behavior is acceptable for shipping.
- Influence model selection, behavior, and tool design to balance quality, cost, latency, and robustness.
Requirements
- 6+ years of professional engineering experience, with ownership of complex systems in production.
- Demonstrated experience owning AI/LLM behavior beyond basic integration, including mitigating variance and failure modes.
- Comfort reasoning about probabilistic systems and tradeoffs (quality vs. cost, recall vs. precision, speed vs. robustness).
- Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.
- Strong product intuition and the ability to define "good" even with non-deterministic outputs.
- Ability to operate independently in ambiguous problem spaces and set quality standards.
- Strong opinions, weakly held, with a focus on rapid iteration and evidence-based adjustments.