Applied Research - Evals & Data
$150k - $300k • Remote • San Francisco • FullTime
Posted 3mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Engineer
About the job
Prime Intellect is building the open superintelligence stack, a full-stack system for post-training at frontier scale, encompassing compute, environments, evaluations, secure sandboxes, high-performance training, and deployment. We are developing open-source models for long-horizon tasks and the platform our research team uses to build them. This role is customer-facing, operating at the intersection of cutting-edge RL/post-training methods, applied data, and agent systems. You will directly influence how advanced models are aligned, evaluated, deployed, and used in the real world by advancing agent capabilities, building robust infrastructure, bridging customer needs with research, and prototyping in the field.
Responsibilities
- Design and iterate on next-generation AI agents for workflow automation, reasoning, and decision-making at scale.
- Refine agent policies, improve reasoning, and enhance reliability and safety using applied data from real deployments.
- Develop distributed systems, evaluation pipelines, and coordination frameworks for reliable, efficient, and large-scale agent operation.
- Build data capture, processing, and versioning workflows for feedback, model traces, and reward signals.
- Translate customer needs and applied data insights into technical requirements for product and research.
- Collaborate with RL and eval teams to ensure real-world signals inform model alignment and reward shaping.
- Rapidly design and deploy agents, evals, and harnesses with customers to validate solutions.
- Use applied evaluation data to iterate on model performance and discover new capabilities.
- Work with customers to understand workflows, data sources, and bottlenecks.
- Prototype agents, data pipelines, and eval harnesses tailored to real use cases.
- Translate customer insights and evaluation results into roadmap and research direction.
- Design and implement novel RL and post-training methods (e.g., RLHF, RLVR, GRPO) for large model alignment.
- Build evaluation harnesses and verifiers to measure reasoning, robustness, and agentic behavior.
- Integrate applied data collection and analytics into the post-training process.
- Prototype multi-agent and memory-augmented systems for customer-facing solutions.
- Rapidly prototype and iterate on AI agents for automation, workflow orchestration, and decision-making.
- Extend and integrate with agent frameworks to support evolving feature requests.
- Architect and maintain distributed training and inference pipelines for scalability and cost efficiency.
- Develop observability and monitoring (e.g., Prometheus, Grafana, tracing) for production deployments.
Requirements
- Strong background in machine learning engineering with experience in post-training, RL, or large-scale model alignment.
- Experience with applied data workflows and evaluation frameworks for large models or agents (e.g., SWE-Bench, HELM, EvalFlow).
- Deep expertise in distributed training/inference frameworks (e.g., vLLM, sglang, Ray, Accelerate).
- Experience deploying containerized systems at scale (Docker, Kubernetes, Terraform).
- Track record of research contributions (publications, open-source contributions, benchmarks) in ML/RL.
- Passion for advancing state-of-the-art in reasoning, measurement, and building practical, agentic AI systems.
Benefits
- Cash Compensation Range of $150-300k
- Equity incentives
- Flexible Work (remote or San Francisco)
- Visa Sponsorship & relocation support
- Professional Development budget
- Team Off-sites & conference attendance