Lead AI Engineer (LLM & Agents)
Remote • Remote - Global • FullTime
Posted 16d ago
About the job
Trust Wallet is seeking a Lead AI Engineer to take ownership of the backend and Agent systems for their LLM-powered products. This end-to-end role involves building Agent workflows, defining quality metrics, ensuring production reliability, and influencing future product direction. The ideal candidate has a proven track record of shipping LLM features to real users, maintaining them post-launch, and demonstrating measurable improvements through data.
Responsibilities
- Design, build, and maintain multi-turn conversation systems, including context management, tool calling, knowledge retrieval (RAG), state management, and human handoff.
- Define success criteria in collaboration with Product, Design, and QA leads.
- Convert production failures into evaluation samples and regression tests.
- Measure the impact and side effects of changes before and after release.
- Diagnose and isolate root causes of failures across the stack (model judgment, knowledge gaps, tool responses, backend instability, conflicting rules).
- Implement appropriate fixes for identified issues, rather than solely relying on prompt adjustments.
- Manage production reliability, including timeouts, retries, graceful degradation, async jobs, and third-party dependency failures.
- Ensure all issues are traceable through logs, all changes are versioned, and all failures have a recovery path.
- Provide clear, reasoned positions on requirements, considering feasibility, cost, risk, and alternatives.
- Propose improved approaches and break down work into phased, shippable deliverables.
- Communicate progress, blockers, ETAs, and next checkpoints proactively.
- Track real-world impact post-launch and provide clear documentation and handover materials.
Requirements
- Production LLM/Agent experience at scale, with features used by 10,000+ active users and post-launch maintenance and improvement.
- Strong backend engineering fundamentals, including owning production code, APIs, data flows, and incident troubleshooting.
- Experience designing and implementing backend services in Golang and integrating with LLMs and AI systems.
- Demonstrated evaluation discipline, including building evaluation sets, regression suites, and production monitoring.
- An ownership mindset, making decisions, surfacing risks early, and ensuring features are verified in production.
- Web3 experience, preferably 1 year in a crypto, DeFi, or Web3 company (wallet, exchange, L1/L2 protocol, or DeFi platform strongly preferred).
Benefits
- Competitive salary and benefits
- Excellent learning and career development opportunities
- Work alongside diverse, world-class talent
- Tackle fast-paced, challenging and unique projects
- Work in a truly global organization with international teams and a flat organizational structure
- Work fully remotely with flexible working hours