Tech Lead, Agent Eval Platform
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
Job Location
Mountain View, CALIFORNIA, United States
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
Moveworks' AI agents are designed to act, plan, and modify enterprise systems, impacting millions of employees. This role focuses on the critical challenge of accurately measuring agent performance across multi-step interactions. You will build the 'judgement layer' of our agent evaluation platform, developing rubrics, judges, and methodologies to create meaningful scores. These scores will not only assess performance but also serve as a reward signal for training agents, enabling them to improve and avoid failures. This is an applied ML role tackling the open research problem of LLMs judging LLMs, specifically within the context of agents interacting with real enterprise environments.
Responsibilities
- Develop the runtime for executing multi-turn agent scenarios, managing the user-agent-world loop, and collecting data for scoring.
- Implement scheduling, retries, and high-concurrency execution for agent evaluations at scale.
- Manage versioned specifications, datasets, and reports, enabling run-to-run comparisons.
- Consolidate existing evaluation workflows into a single orchestration service.
- Establish reliability standards and Service Level Objectives (SLOs) for the evaluation harness.
- Enable self-service evaluation capabilities for all teams.
- Lead the adoption of OpenTelemetry for agent platform observability.
- Define the span data model for agent trajectories to enable queryable trace data.
- Implement trace context propagation across asynchronous operations and long-lived sessions.
- Ensure full prompts and completions are preserved and eval traffic does not contaminate data.
- Develop fault attribution and cross-run diffing capabilities.
- Build the debug surface and tracing contract for harness engineers.
- Develop the simulation environment for enterprise systems, including stateful fakes and datastore persistence.
- Implement per-run data injection and programmatic setup/teardown for hermetic and repeatable runs.
- Create LLM-driven user simulators and scripted state-machine simulators.
- Implement contract testing for mocks against real API schemas.
- Develop isolated sandbox environments for agent testing.
- Lay the foundation for using evaluation signals to optimize agent performance.
Requirements
- Experience in at least 3 of the following areas: distributed systems (idempotency, delivery guarantees, isolation, determinism, reproducibility), orchestration and workflow runtimes (DAG execution, scheduling, retries, backfills, high-concurrency job systems), observability internals (OpenTelemetry SDKs/collectors, semantic conventions, span context propagation, high-cardinality trace data), concurrent and async programming (Python asyncio, Go concurrency, structured cancellation), data-intensive pipelines (high-volume ingest, schema evolution, sampling/retention), gRPC/protobuf service and interface design.
- Ability to tech lead other engineers and manage end-to-end project delivery.
- Strong communication and soft skills.
- 10+ years of experience building production backend or infrastructure systems.
- Proficiency in Python or Go (ideally both).
- Experience designing and operating systems that handle real traffic at scale.
- Comfort in making non-deterministic systems measurable.
- Interest in transforming fuzzy agent behavior into a quantifiable signal for release gating.
- Comfort with ambiguity and novel problems without textbook solutions.