Staff Software Engineer, Agent Eval Platform
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
Job Location
Mountain View, CALIFORNIA, United States
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
Moveworks' AI agents are designed to act, plan, call tools, and modify enterprise systems, impacting millions of employees. The core challenge for this team is developing a precise measurement system to evaluate agent performance across multi-step interactions. This role will build the "judgement layer" of the agent evaluation platform, focusing on rubrics, judges, and calibration against human labels to create a meaningful scoring methodology. The developed signal will not only score agent actions but also serve as a reward signal for training, enabling agents to improve their performance. This is an applied ML role focused on the novel problem of LLMs evaluating LLMs, particularly in the context of agents interacting with real-world enterprise systems.
Responsibilities
- Develop the runtime for executing multi-turn agent scenarios end-to-end, including environment setup, user simulation, and data collection.
- Implement scheduling, retries, and high-concurrency execution for production-scale datasets.
- Manage versioned specifications, datasets, and reports with built-in run-to-run comparison.
- Consolidate existing one-off eval workflows into a single orchestration service.
- Establish reliability standards and Service Level Objectives (SLOs) for the evaluation harness.
- Enable self-service evaluation capabilities for all teams.
- Lead the integration of OpenTelemetry for agent platform observability.
- Design the span data model for agent trajectories to enable queryable trace data.
- Implement trace context propagation across asynchronous boundaries and long-lived sessions.
- Ensure full prompts and completions are preserved throughout the pipeline.
- Develop fault attribution and cross-run diffing capabilities.
- Build debug surface support and define the tracing contract for agent development.
- Create stateful simulation environments for enterprise systems (ITSM, HR, knowledge bases).
- Implement per-run data injection and programmatic setup/teardown for hermetic and repeatable runs.
- Develop LLM-driven user simulators and scripted state-machine simulators.
- Integrate contract testing for mocks against real API schemas in CI.
- Lay the foundation for using evaluation signals to optimize agent performance.
Requirements
- Experience in distributed systems, focusing on idempotency, delivery guarantees, isolation, determinism, and reproducibility.
- Experience with orchestration and workflow runtimes (e.g., Temporal, Airflow, Argo).
- Experience with observability internals as a builder (e.g., OpenTelemetry SDKs, collectors, span context propagation).
- Proficiency in concurrent and asynchronous programming (e.g., Python asyncio, Go concurrency).
- Experience with data-intensive pipelines, including high-volume ingest and schema evolution.
- Experience with gRPC/protobuf service and interface design.
- 8+ years of experience building production backend or infrastructure systems.
- Strong proficiency in Python or Go (ideally both).
- Experience designing and operating systems that handle real traffic at scale.
- Comfort in making non-deterministic systems measurable.
- Comfort with ambiguity and novel problem-solving.