Member of Technical Staff (Evals & Post-Training)
$165k - $325k • Remote • New York • FullTime
Posted 1mo ago
Job Location
New York
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Ambral Labs is building a platform to help enterprises leverage their historical data to improve AI model performance. The system reconstructs past company workflows into replayable environments, allowing for the evaluation and enhancement of agent behavior through reinforcement learning and other post-training techniques. This enables companies to develop better, more cost-efficient AI models that they own and control, rather than relying on third-party providers. Having graduated from YC and secured significant funding, Ambral Labs is now expanding its founding team.
Responsibilities
- Build an environment factory to convert enterprise data into runnable environments.
- Design graders to translate business objectives into verifiable rewards.
- Develop methods for extracting tasks, trajectories, and evaluation cases from historical workflows.
- Create representative, reproducible, and overfitting-resistant eval sets.
- Identify optimal combinations of models, tools, context, and policies to maximize performance and reduce inference cost.
- Train and evaluate agents operating over long horizons, incomplete information, and large tool spaces.
- Build replay and observability systems for explainable and measurable agent behavior.
- Scale systems from individual environments to thousands of concurrent training and evaluation runs.
Requirements
- 1-7 years of experience building production software or machine-learning systems.
- Experience with reinforcement-learning environments, LLM post-training, evaluation infrastructure, or agent harnesses is a plus.
- Understanding of the interaction between environment design, reward design, context, tooling, and policy behavior.
- Ability to translate ambiguous business objectives into reliably evaluable tasks and signals.
- Skill in diagnosing model limitations across model, context, tools, harness, or training.
- Proficiency in moving between research questions and production implementation.
- Strong software engineering skills for building systems that process large, messy datasets at scale.
- Commitment to reproducibility, observability, and understanding model behavior.
Benefits
- Significant equity and ownership
- Equinox membership
- Free meals, coffee, and snacks
- Health insurance
- Unlimited PTO