Research Lead - Pre-training Safety
$290k - $450k • Berkeley Office • Full-time
Posted 1mo ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Research Engineer
About the job
FAR.AI, a non-profit AI research institute dedicated to ensuring advanced AI is safe and beneficial, is seeking a Research Lead to spearhead its pre-training safety initiatives. This role focuses on shaping AI models at their source by controlling capabilities and internal representations, aiming to prevent misuse and reduce loss-of-control risks. The work involves scaling methods for large models, validating approaches like data filtering and gradient routing, and exploring techniques such as unlearning. The Research Lead will direct these efforts, collaborate with red teams for stress-testing, and analyze scalability to frontier systems. This position offers significant autonomy within an impact-driven environment, focusing on empirically grounded, scalable ML safety research.
Responsibilities
- Define and lead a research workstream end-to-end, articulating a research agenda with a clear theory of change for mitigating catastrophic risks or increasing AI upside.
- Grow and lead a team of technical staff, mentoring junior members in research skills and ML engineering.
- Lead novel research projects, potentially with unclear markers of progress or success.
- Share research findings through publications and presentations to drive adoption and change.
- Contribute to the FAR.AI intellectual environment and research culture.
- Build a research field around the agenda through grantmaking and events, and connect it to real-world deployments.
- Scale methods like Deep Ignorance by over an order of magnitude (>100B parameter models with >1T tokens).
- Partner with red teams to stress-test resulting models and analyze scalability to frontier systems.
- Explore techniques such as gradient routing and unlearning.
- Develop improved data filtering methods, potentially using data attribution or sophisticated classifiers.
- Train models to actively remove harmful capabilities, possibly through interleaving next-token prediction with unlearning.
- Add synthetic data to pre-training or mid-training to shape model representations and behavior.
Requirements
- Experience in leading research projects and teams.
- Strong understanding of AI safety research, particularly pre-training safety.
- Ability to articulate a research agenda with a clear theory of change.
- Experience in mentoring junior team members.
- Ability to write code and run experiments.
- Desire to work on impactful research directions with mission-driven colleagues.
- Interest in pursuing empirically grounded, scalable research directions.
- Ability to speak freely and contribute to a culture of open discussion.