Member of Technical Staff - Pre-Training Evals Infra
New York, NY • FullTime
Posted 4d ago
Job Location
New York, NY
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Reflection is a research lab dedicated to making intelligence open and accessible. We build open models that empower users to control their intelligence and shape the future of AI. In this role, you will develop the critical evaluation infrastructure that researchers rely on to understand model capabilities, inform training decisions, and assess readiness for release. Your work will encompass evaluation authoring, distributed execution, and results analysis, with a specific emphasis on pre-training. You will collaborate closely with researchers to enhance the speed, reliability, and ease of use of evaluations, ensuring that the outcomes accurately reflect the models being assessed.
Responsibilities
- Design and evolve evaluation abstractions, building clear APIs and data models for evaluation tasks, datasets, configurations, model outputs, and scores.
- Build scalable and reliable distributed execution infrastructure for evaluating model checkpoints throughout training and before release.
- Optimize evaluation and inference performance to improve throughput, GPU utilization, and time to results.
- Establish evaluation correctness and reproducibility through validation tests, versioning of datasets and configurations, and provenance tracking.
- Develop researcher-facing tools, including interfaces and dashboards for inspecting model outputs and comparing experiments.
- Operate and monitor evaluations as training progresses, tracking workloads, triaging failures, and developing alerts.
- Translate researcher needs into reusable, self-service systems, identifying bottlenecks and prioritizing improvements.
Requirements
- Strong proficiency in Python, C++, or Rust, with a deep understanding of concurrency, memory behavior, and performance.
- Experience designing, building, and operating reliable, maintainable distributed systems.
- Working knowledge of model inference and evaluation, including the impact of implementation and configuration choices on measured model quality.
- Ability to investigate complex problems across system boundaries and identify root causes.
- Strong product judgment and experience translating technical user needs into clear APIs and usable tools.
- Comfort owning ambiguous problems end-to-end, prioritizing competing needs, and balancing delivery with long-term reliability.
- Attention to detail and persistence in investigating subtle discrepancies.
Benefits
- Top-tier compensation: Salary and equity structured to recognize and retain talent.
- Stock options
- Comprehensive health, dental, vision, and life insurance
- Annual wellness allowance
- Provided lunch and dinner in the office daily
- 22 weeks paid parental leave
- Unlimited paid time off in the U.S.
- 30 days paid time off in the U.K.
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations