Staff Software Engineer, Agent Eval Platform
Remote • Santa Clara, CALIFORNIA, United States • Full-time
Posted 1mo ago
Job Location
Santa Clara, CALIFORNIA, United States
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
Moveworks' AI agents are designed to act, plan, and modify enterprise systems, impacting millions of employees. This role focuses on building the core measurement and evaluation platform for these agents. You will develop the 'judgement layer,' creating rubrics, judges, and methodologies to accurately score agent performance across multi-step interactions. This score will not only provide insights into agent failures but also serve as a crucial reward signal for training and improving agent behavior. This is an applied machine learning role tackling the complex, open research problem of LLMs evaluating LLMs, specifically within the context of agents performing real-world actions in enterprise environments.
Responsibilities
- Develop the runtime for executing multi-turn agent scenarios, managing the agent-world interaction loop, and collecting data for validation and scoring.
- Implement scheduling, retries, and high-concurrency execution for production-scale datasets.
- Establish versioned specifications, datasets, and reports with built-in run-to-run comparison capabilities.
- Consolidate existing one-off eval workflows into a unified orchestration service.
- Define and maintain reliability standards and Service Level Objectives (SLOs) for the evaluation harness.
- Enable self-service evaluation capabilities for all teams.
- Lead the integration of OpenTelemetry for agent platform observability, replacing existing logging and correlation mechanisms.
- Design the span data model for agent trajectories to enable queryable trace data.
- Implement trace context propagation across asynchronous boundaries for long-lived sessions.
- Ensure full prompts and completions are preserved throughout the pipeline and prevent eval traffic from contaminating data.
- Develop fault attribution and cross-run diffing capabilities to identify component failures and changes.
- Build the debug surface and tracing contract for harness engineers and agent development teams.
- Create stateful simulation environments that mimic enterprise systems (ITSM, HR, knowledge bases) using real datastores.
- Implement per-run data injection and programmatic setup/teardown for hermetic and repeatable runs.
- Develop LLM-driven user simulators for open-ended personas and scripted simulators for deterministic flows.
- Integrate contract testing for mocks against real API schemas in CI to ensure simulation fidelity.
- Lay the foundation for using evaluation signals to optimize agent performance.
Requirements
- 8+ years of experience building production backend or infrastructure systems.
- Strong proficiency in Python or Go, ideally both.
- Experience designing and operating systems that handle real traffic at scale.
- Comfort with making non-deterministic systems measurable and turning fuzzy behavior into actionable signals.
- Comfort with ambiguity and novel problems without textbook solutions.
- Experience in at least 3 of the following areas: distributed systems (idempotency, delivery guarantees, isolation, determinism, reproducibility), orchestration and workflow runtimes (DAG execution, scheduling, retries, backfills, high-concurrency job systems), observability internals (OpenTelemetry SDKs and collectors, semantic conventions, span context propagation, high-cardinality trace data), concurrent and async programming (Python asyncio, Go concurrency, structured cancellation), data-intensive pipelines (high-volume ingest, schema evolution, sampling and retention trade-offs), gRPC/protobuf service and interface design.