Agent Evaluation and Self-Improvement Engineer
$224k - $431k • Remote • US, CA, Santa Clara
Posted 3d ago
Job Location
US, CA, Santa Clara
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Agent Engineer
About the job
This role focuses on establishing and implementing a comprehensive evaluation strategy for autonomous agents at NVIDIA. The goal is to ensure that changes to agents, their underlying models, harnesses, tools, and skills lead to improvements rather than regressions. You will design systems that allow agents to leverage evaluation results for self-improvement, building trust in the process of autonomous evolution. This involves creating robust evaluation frameworks, mining production data for insights, and developing realistic simulation environments for safe, end-to-end agent task execution.
Responsibilities
- Build consistent evaluation across agent harnesses with automated regression checks.
- Mine production traces to identify failures, new usage patterns, and cost/latency issues, converting them into evaluation cases and improvement signals.
- Evaluate the quality and coverage of evaluation datasets against production usage.
- Develop self-improvement loops where agents propose and test changes to their own configurations and skills within sandboxed environments.
- Create realistic enterprise environments for agents to perform long, end-to-end tasks safely.
- Measure the performance of multi-agent swarms, including work splitting, handoffs, and permission adherence.
- Calibrate model and agent judges against human labels to ensure evaluation honesty and detect cheating.
- Collaborate with product, platform, and hardware teams, and contribute to industry evaluation standards.
Requirements
- BS or MS in Computer Science, Engineering, or related field (or equivalent experience).
- 12+ years of experience building large-scale systems, including ML or agent evaluation systems and their data pipelines.
- Hands-on experience building and evaluating agents on an agent harness.
- Solid understanding of experiment design, including statistical significance, variance, and A/B testing.
- Experience setting technical direction across teams as a senior individual contributor.
- Proficiency in Python, Go, Rust, or similar programming languages.
Benefits
- Equity
- Benefits