New: upload your resume and get matched to AI jobs. Free to start →

Software Engineering Director, Agentic Evaluations

$260k - $320k • Remote • Remote (US) • Full-time

Posted 1mo ago

Remote Work Policy

Fully remote

Employment Type

Full-time

Categories

AI Agent Engineer

About the job

G2 is the world's largest and most trusted software marketplace, now combined with Capterra, SoftwareAdvice, and GetApp to create a leading source of software insights. We are transforming the B2B software industry by providing trusted data foundations for the age of AI. As a Software Engineering Director, Agentic Evaluations, you will lead the delivery of credible evaluations for software vendor agents, enhancing the system that validates agent performance and leading the core team behind it. You will apply agentic engineering techniques to productionize your work.

Responsibilities

  • Scope the feasibility and effort for evaluating vendor agents, identifying practical paths to credible evaluations.
  • Own the full stack of the core agent evaluation system, from admin and API surface to workflow dispatch.
  • Design evaluation system primitives that generalize across software verticals for reuse.
  • Distill repeatable processes for onboarding and maintaining integrations into AI skills or agents to increase evaluation velocity.
  • Stay updated on emerging agent evaluation frameworks and techniques for improvements and industry-accepted distribution of results.
  • Collaborate with data science peers on developing proprietary benchmarks.
  • Facilitate the growth and mentorship of a core engineering team, providing technical leadership and subject matter expertise in evaluations.
  • Socialize the use of evaluations in agent-oriented products across the organization.

Requirements

  • 10+ years of professional programming experience in backend or full-stack environments.
  • 2+ years of direct experience managing engineers.
  • Expert-level proficiency in backend development using languages like Python, Java/Kotlin, Typescript/Javascript, or Go.
  • Strong proficiency in associated backend frameworks like FastAPI or Node.js.
  • Direct experience creating evaluations against customer-facing agents using trajectory trace data and rubrics to measure task completion, accuracy, correctness, or policy adherence.
  • Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications.
  • Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi in daily development workflows.
  • BS/BA degree in a related field.