Senior Software Engineer, Open Harness Engineering
$184k - $357k • Remote • US, CA, Remote
Posted 1d ago
Job Location
US, CA, Remote
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Agent Engineer
About the job
We are seeking a Senior Software Engineer to contribute to the development of core libraries, evaluation systems, and reusable harness capabilities for AI agents. This role involves hands-on work with open-source agent harnesses, sandboxed execution, model-provider infrastructure, and agent benchmarks to enhance the next generation of agents. You will analyze agent evaluation data to drive improvements, diagnose failures, contribute to upstream projects, and demonstrate gains in quality, reliability, cost, and latency across various models. Join our team to build foundational technology for future autonomous software systems.
Responsibilities
- Define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmark environments.
- Build and operate scalable, reproducible evaluation systems including CI, scheduled compute, sandboxed execution, artifact provenance, trace capture, and analysis.
- Diagnose agent failures from traces, logs, tool results, and benchmark artifacts, identifying root causes across harness, evaluator, environment, inference, and model components.
- Design controlled experiments and promotion criteria to differentiate genuine harness improvements from noise, benchmark artifacts, or model-specific gains.
- Ship focused harness and runtime improvements, including upstream open-source contributions, detectors, regression tests, and maintainable documentation.
- Collaborate with Relay, Hermes, model, infrastructure, and research teams to translate recurring optimization needs into reusable runtime, trace, and evaluation capabilities.
Requirements
- Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Applied Math, or a related field, or equivalent experience.
- 8+ years of hands-on software engineering experience, with demonstrated technical ownership of production systems and architecture design leadership.
- Expert Python skills and working proficiency in Rust, Go, C++, or TypeScript, with the ability to debug systems across language and process boundaries.
- Experience building or extending LLM agents, coding agents, tool-use loops, model-provider integrations, developer tools, or evaluation infrastructure.
- Solid understanding of asynchronous execution, subprocesses, containers, sandboxing, callbacks, retries, networking, and distributed compute.
- Experience crafting reproducible experiments, benchmark methodology, performance investigations, regression gates, or reliability analysis under nondeterministic conditions.
- A record of shipping and maintaining open-source or developer-facing software with sound testing, API development, documentation, and code reviews.
Benefits
- Highly competitive salaries
- Comprehensive benefits package
- Eligibility for equity