Software Engineer, Coding Evaluation & Training Data
Remote • United States - Remote • FullTime
Posted 2mo ago
About the job
Surge is seeking a Software Engineer to focus on Coding Evaluation & Training Data. This role is at the intersection of software engineering and product, responsible for building and running systems that train frontier models to code. You will own end-to-end coding data projects for top AI labs, designing tasks, RL environments, and evaluation schemes that mirror real-world software engineering practices. This is an ideal opportunity for a software engineer passionate about training data quality, agentic evaluation, and system design involving humans, models, and tools, rather than traditional product feature development.
Responsibilities
- Own end-to-end coding data projects from scoping and pilot design through execution, iteration, and scale-up.
- Design agentic training workflows and task structures that mirror real-world software engineering tasks like refactoring, debugging, code review, and large-repo navigation.
- Define and iterate on rubrics, golden sets, and reward signals to accurately capture engineering value.
- Evaluate data and worker output with strong engineering judgment to ensure quality for frontier training.
- Design and run qualification processes for coding workers, including hands-on coding ability assessments.
- Set up or partner on complex technical environments such as containers, repos, test harnesses, sandboxes, and code execution infrastructure.
- Partner with client technical staff to translate high-level training goals into concrete projects and technical environments.
- Collaborate with internal engineering, product, and operations teams to enhance coding data products, internal tools, and execution processes.
Requirements
- 2–6+ years of professional software engineering experience building and maintaining real systems.
- Strong coding ability in at least one mainstream language and experience with production codebases.
- High regard for good engineering practices, including correctness, code quality, and understanding of how real engineering teams operate.
- Ability to reason about and debug technical environments, including containers, dependencies, and automated test setups.
- Interest in end-to-end project ownership, encompassing scoping, workflow design, execution, and continuous improvement.
- Excellent written and verbal communication skills, capable of credible discussions with senior client engineers and translating fuzzy goals into actionable plans.
- Enthusiasm for AI/ML systems and the impact of data, evaluation, and reward design on agentic coding capabilities.