Research Engineer - Data and Evaluation
$150k - $350k • Remote • San Francisco • FullTime
Posted 1d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Research Engineer
About the job
Prime Intellect is building the open superintelligence stack, a full-stack system for post-training at frontier scale. Our platform, Lab, unifies compute, environments, evaluations, secure sandboxes, high-performance training, and deployment. We are developing open-source models trained end-to-end for long-horizon tasks and the platform our research team uses to build them. This role is for someone who wants to build at the intersection of frontier research, real infrastructure, and go-to-market for a new category.
As a Research Engineer, you will build the tasks and environments that train our models through RL, and the evaluations that measure their capabilities. You will collaborate directly with researchers to diagnose model failures, examining both task implementation and success scoring.
Responsibilities
- Design and build challenging evaluations for coding and long-horizon agents.
- Build executable environments in Verifiers with realistic task setup and clear success criteria.
- Develop and test scoring logic and verifiers, investigating reward hacks and misleading progress measures.
- Trace failures through model outputs and harness code to distinguish model limitations from environment or scoring issues.
- Build and scale RL data pipelines for generating and curating challenging coding and agentic tasks.
- Collaborate with RL researchers to assess if changes to tasks and training data improve performance on held-out tasks.
Requirements
- Strong software engineering skills in Python and experience building data pipelines, evaluation systems, or ML workflows.
- Hands-on experience with language models or agents, understanding how training data and evaluation choices impact model behavior.
- Ability to inspect complex data and model outputs, identify failure modes, and translate ambiguous quality questions into testable measurements.
- Proven record of owning technical projects through experimentation, implementation, and validation, while collaborating with researchers and engineers.
- Hands-on experience building technical evaluations or agent environments, with engineering depth to debug harness and scoring code.
- Experience with large-scale synthetic data generation, dataset curation, or distributed processing for model training.
- Experience with evaluation methodology, executable coding tasks, agent environments, verifiers, or reward design.
- Experience with experiment analysis, statistical testing, contamination detection, and reproducible research tooling.
- Experience with RL or post-training data, including task difficulty selection and analysis of rollout or reward distributions.
- Contributions to Verifiers, Prime-RL, or comparable environment and evaluation frameworks.
- Work on model-based grading, long-horizon coding agents, or diagnosing reward exploitation.
Benefits
- Cash Compensation Range of $150-350k
- Equity incentives
- Flexible work arrangements (remote or in-person)
- Visa sponsorship
- Relocation assistance
- Quarterly team off-sites
- Hackathons
- Conferences and learning opportunities