Senior Software Engineer, AI Evaluation Infrastructure
Remote • Global - Remote
Posted 1d ago
Job Location
Global - Remote
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
Turing is seeking a Senior Software Engineer to join their AI Evaluation Infrastructure team. This role focuses on building the core infrastructure that frontier AI labs use to evaluate coding agents and language models. You will be responsible for designing and implementing features across the entire stack, from data models and background jobs to user-facing interfaces. The ideal candidate is an experienced, independent engineer who can translate loosely defined requests into robust, tested, and reliable solutions. This position offers the opportunity to work on cutting-edge AI evaluation tools and contribute to the advancement of AI capabilities.
Responsibilities
- Design and implement features end-to-end, including data models, APIs, background workers, and UI components.
- Develop and enhance systems for evaluating agents and models, including running trials in sandboxed environments, scoring results, and aggregating data.
- Improve the quality checking of benchmark tasks using verifiers, rubric-based grading, and LLM-as-a-judge approaches.
- Integrate new models, agents, and benchmarks in a generalized manner.
- Ensure system performance and reliability as usage scales, focusing on efficient queries, safe migrations, and concurrency.
- Review teammate pull requests and maintain codebase consistency according to engineering guidelines.
Requirements
- Typically 5+ years of experience building and operating production backend systems, with end-to-end ownership of at least one service.
- Strong proficiency in Python, with experience in Django (or similar framework), PostgreSQL, and a task queue like Celery.
- Hands-on experience in LLM evaluation beyond basic API calls, understanding benchmark task construction, agent trial scoring, and result aggregation.
- Demonstrated judgment in assessing task quality, identifying ambiguous instructions, weak verifiers, and potential biases.
- Ability to make sound trade-offs, favoring minimal designs and extending existing solutions.
- Strong sense of ownership, capable of taking fuzzy requests to completion, including tests and edge cases.
- Quick adoption of team conventions for naming, structure, testing, and error handling.
- Comfort with TypeScript and React for frontend changes.
- Clear and concise writing skills for pull request descriptions and design notes.