Research Engineer, Coding Evaluation & Training Data
Remote • United States - Remote • FullTime
Posted 2mo ago
About the job
As a Research Engineer, Coding Evaluation & Training Data at Surge, you will be at the forefront of developing systems that train frontier models to code. This role blends software engineering and product development, focusing on creating and managing the processes that teach AI to perform real-world software engineering tasks. You will own coding data projects from conception to completion, designing tasks, environments, and evaluation methods that accurately reflect the complexities of software development. This is an excellent opportunity for a software engineer passionate about training data quality, agentic evaluation, and system design involving humans, models, and tools.
Responsibilities
- Own end-to-end coding data projects, from initial scoping and pilot design through execution, iteration, and scale-up.
- Design agentic training workflows and task structures that mirror real-world SWE work (e.g., refactoring, debugging, code review, large-repo navigation, tool use).
- Define and iterate on rubrics, golden sets, and reward signals that capture true engineering value.
- Evaluate data and worker output with strong SWE taste, and make calls about what meets the bar for frontier training.
- Design and run qualification processes for coding workers, including hands-on assessments of their coding ability.
- Set up or partner on complex technical environments (e.g., containers, repos, test harnesses, sandboxes, code execution infrastructure).
- Partner with technical staff at our clients to translate high-level training goals into concrete projects and technical environments.
- Collaborate closely with Surge engineering, product, and operations to improve our coding data products, internal tools, and execution processes.
Requirements
- 3–6+ years of professional software engineering experience building and maintaining real systems.
- Strong coding ability in at least one mainstream language and comfort working in production codebases.
- High taste for good engineering: care about correctness, code quality, and how real engineering teams actually work.
- Ability to reason about and debug technical environments, including containers, dependencies, and automated test setups.
- Interest in owning projects end-to-end, including scoping, workflow design, execution, and continuous improvement.
- Excellent written and verbal communication skills, with the ability to speak credibly with senior client engineers and translate fuzzy goals into actionable plans.
- Excitement about AI/ML systems and the role of data, evaluation, and reward design in improving agentic coding capabilities.