Software Engineer, Evaluation Platform / Infra
$300k - $350k • San Francisco • FullTime
Posted 1mo ago
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Engineer
About the job
Thinking Machines is building AI that extends human will and judgment, and we're looking for individuals passionate about creating a human-centric future. In this role, you will design and build an end-to-end evaluation platform crucial for developing frontier AI systems. This platform will empower researchers to easily author, run, and analyze model evaluations at scale, supporting both standardized benchmarks and internal research needs. You will collaborate closely with various teams to enhance how we evaluate models and translate results into research decisions.
Responsibilities
- Design, build, and maintain the platform for authoring, running, tracking, and analyzing model evaluations.
- Develop capabilities end-to-end across evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications.
- Create flexible abstractions for evaluation tasks, environments, graders, datasets, and model outputs.
- Ensure evaluation results are reproducible and trustworthy through versioning, provenance, observability, and quality controls.
- Partner with researchers to identify bottlenecks and transform bespoke workflows into self-serve systems.
- Collaborate with the Research Tooling engineering team.
Requirements
- Bachelor's degree in computer science, engineering, machine learning, or related field, or equivalent practical experience.
- Two years of post-grad work experience as a software engineer or ML engineer (excluding internships).
- Hands-on experience building or maintaining evaluations, benchmarks, graders, or model-quality systems for large language or multimodal models.
- Strong software engineering fundamentals and experience building reliable, maintainable systems.
- Proficiency in at least one backend programming language (Python and Rust primarily used).
- Experience with React and Typescript on the frontend.
- Experience with databases, data pipelines, distributed systems, or other data-intensive infrastructure.
- Comfort working across the full stack and owning projects from discovery to operation.
- Experience collaborating with cross-functional partners and subject-matter experts.
- Experience building frameworks, SDKs, or developer tools with thoughtful abstractions and strong UX (preferred).
- Experience with distributed job execution, workflow orchestration, sandboxed environments, or large-scale data processing (preferred).
- Experience building intuitive interfaces for inspecting complex data, comparing experiments, or debugging model behavior (preferred).
- Familiarity with large language or multimodal model evaluation, including model-based grading, human evaluation, or synthetic data (preferred).
- Experience working closely with researchers to understand workflows and translate evolving needs into durable systems (preferred).
- Experience at a startup or on a small team building complex products end-to-end (preferred).
Benefits
- Generous health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support