Member of Technical Staff - Research Software Engineer - Safety Evaluations Infrastructure
New York, NY • FullTime
Posted 3d ago
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Reflection is seeking a Research Software Engineer for its Safety team to design, build, and own the infrastructure for sensitive model evaluations, including those in CBRN, child safety, and dangerous-capability domains. This role is critical for informing model release decisions, requiring systems that are secure, isolated, reproducible, and trustworthy. It's a deeply technical position at the intersection of platform engineering, security, and safety research, involving close collaboration with domain experts, legal, and safety researchers to develop robust, scalable infrastructure. This includes creating sandboxed execution environments, controlled data pipelines for sensitive material, access controls, audit logging, and tooling for researchers to safely measure model capabilities in high-consequence areas.
Responsibilities
- Design and build secure, sandboxed infrastructure for sensitive model evaluations.
- Develop controlled data pipelines and storage for sensitive material with strict access controls and safeguards.
- Translate evaluation designs from safety researchers and domain experts into reliable, reproducible systems.
- Build tooling for high-throughput evaluations in isolated environments.
- Implement infrastructure for measuring AI capability uplift in high-consequence domains.
- Develop and implement guardrails, monitoring, and compartmentalization for sensitive work.
- Write production-quality Python for data processing and evaluation systems.
- Enhance the reliability, security, and developer experience of the evaluation platform.
Requirements
- Strong software engineering skills, particularly in Python, with experience building reliable, scalable infrastructure or platform systems.
- Experience building sandboxed, isolated, or security-sensitive execution environments.
- Solid understanding of security engineering fundamentals (least privilege, RBAC, secrets management, encryption, audit logging, defense-in-depth).
- Experience building data pipelines and handling sensitive data with safeguards.
- Ability to own problems end-to-end, including ambiguous and cross-functional ones.
- Comfort working on sensitive projects requiring discretion and integrity.
- Experience building evaluation, benchmarking, or experimentation infrastructure for ML systems (preferred).
- Experience working with LLMs, agents, or ML training/inference pipelines (preferred).
- Familiarity with dangerous-capability or dual-use domains and their security considerations (preferred).
- Familiarity with compliance frameworks for sensitive data handling (preferred).
Benefits
- Top-tier compensation (salary and equity)
- Stock options
- Comprehensive medical, dental, vision, and life insurance
- Annual wellness allowance
- Provided lunch and dinner in the office
- 22 weeks paid parental leave
- Unlimited paid time off (US) / 30 days paid time off (UK)
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations