Evals Jobs

LangChain is seeking a Fullstack Engineer to join the LangSmith team, focusing on our commercial AI observability and evaluation platform. This role offers the chance to develop new features and functionality, collaborating closely with enterprise customers, developer end-users, and internal stakeholders. You will contribute to a product that is shaping the future of AI agents in the real world, leveraging a strong foundation in open-source tools and a rapidly growing platform.

$145k - $180k

OpenAI

We are seeking a Machine Learning Engineer to develop and productionize new classification and evaluation capabilities. This role involves translating open-ended questions about model behavior into well-defined ML problems, building effective methods, and rigorously assessing their reliability. You will own classifiers throughout their lifecycle, from supervised fine-tuning and distillation to evaluation, deployment, and production inference. Collaborating closely with researchers to identify valuable problems and with software engineers to scale promising approaches will be key to success.

$381k - $510k

Reflection ai

Reflection is a research lab dedicated to making intelligence open and accessible. We build open models that empower users to control their intelligence and shape the future of AI. In this role, you will develop the critical evaluation infrastructure that researchers rely on to understand model capabilities, inform training decisions, and assess readiness for release. Your work will encompass evaluation authoring, distributed execution, and results analysis, with a specific emphasis on pre-training. You will collaborate closely with researchers to enhance the speed, reliability, and ease of use of evaluations, ensuring that the outcomes accurately reflect the models being assessed.

Dodo is seeking a Senior ML Engineer to enhance the intelligence, speed, and reliability of healthcare's front desk agents. This role involves owning the models, evaluation, and training loops that process millions of real conversations daily, continuously improving the system based on real-world interactions. You will be instrumental in making Dodo's agents smarter and more effective in handling patient communications.

Anthropic

Anthropic's mission is to create reliable, interpretable, and steerable AI systems, believing that powerful AI can accelerate scientific progress. The Claude Science team is building an AI workbench that consolidates tools for researchers, enabling them to manage literature, databases, scientific computing, analysis, and output generation within a single environment. This role offers the opportunity to shape a new product category, influencing both the product and its architecture. You will act as a technical leader, focusing on the end-to-end researcher experience, collaborating with internal research teams to integrate model capabilities into production, and taking ownership of future product development.

$405k - $485k

Anthropic

Anthropic is seeking a Manager of Applied AI Engineering to lead a team of technical engineers embedded with their largest and most strategic accounts. These partner-customers are key revenue drivers and distribution channels, utilizing Claude to transform their businesses. The team will work hands-on with these organizations, co-designing technical solutions, advising on architecture, developing evaluation frameworks, and guiding advanced implementation patterns for LLMs. The manager will be responsible for hiring, coaching, and growing the team while maintaining technical oversight and setting high engineering standards. This role involves close collaboration with various internal teams and external stakeholders to drive technical impact and shape Anthropic's product and research roadmap.

$320k - $405k

We are an applied AI lab building end-to-end software agents, known for creating Devin, the first AI software engineer. Our team comprises world-class talent from leading AI companies. We are seeking a founding member for our safety team to tackle critical challenges in AI safety and alignment. This role is hands-on, focusing on ensuring our AI agents behave correctly and safely in real-world applications, rather than solely in research benchmarks. You will play a pivotal role in shaping the safety and alignment of our AI systems.

Asana's AI organization is building the platform and products that integrate AI into how teams work. This includes AI Teammates, an orchestration layer for agents, and the AI platform supporting all Asana AI features, covering model access, retrieval, context, evaluation, and production quality. We are seeking a Senior Staff Software Engineer to lead technical direction across this organization, tackling complex cross-team challenges, making critical architectural decisions, and shaping the future of AI within Asana's products. This role involves influencing strategy across multiple engineering teams located in San Francisco, New York, Vancouver, and Warsaw.

$297k - $350k

LangChain is seeking a Senior Fullstack Engineer to join the team building LangSmith, their commercial observability and evals platform. This role offers the opportunity to significantly influence the platform's technical direction while collaborating with enterprise customers, developer end-users, and internal stakeholders. You will be instrumental in shaping the future of AI agent engineering by contributing to a product used by a significant portion of the Fortune 500.

$175k - $240k

Baseten powers mission-critical inference for leading AI companies by uniting applied AI research, flexible infrastructure, and seamless developer tooling. We enable companies at the frontier of AI to bring cutting-edge models into production. This role focuses on the applied end of post-training research, working directly with stakeholders from fast-growing AI companies to post-train open-source models that outperform frontier closed models on specialized tasks. The day-to-day involves extracting signal from complex, domain-specific datasets and building reward functions, environments, eval harnesses, and training pipelines to improve models that ship to production and reach millions of users.

$200k - $275k

Knowtex is seeking an ML Engineer / Researcher focused on Large Language Models to contribute to the next generation of their AI stack. This role involves developing and optimizing models for clinical documentation and structured clinical reasoning, aiming to enhance quality, cost, latency, and control. The position is research-heavy with a direct path to production, requiring the design of experiments, building datasets and evaluation systems, and fine-tuning and post-training models. The successful candidate will collaborate closely with engineering and clinical teams to deploy solutions at scale and will play a key role in shaping Knowtex's long-term LLM strategy.

Snorkel AI is seeking a Research Scientist to advance the creation of high-quality data and environments for AI agents. You will build and optimize pipelines that integrate real-world data, automated generation, and human expert input. Collaborating with domain experts, academic partners, customers, and internal teams, you will scale these pipelines to address frontier model performance gaps and enhance data and environment diversity. Your work will empower human knowledge and judgment, enabling experts to refine data and agentic environments, solidifying Snorkel's position as a leader in frontier data solutions. This role is perfect for individuals passionate about advancing frontier AI through data and environment creation, and who enjoy translating research into scalable, reusable systems.

$200k - $375k

Perplexity is seeking an experienced Machine Learning Engineer to advance how AI systems search, reason, and work together to solve complex problems. Our work spans search and retrieval, LLM post-training, multi-agent training, and the harnesses that make these systems effective. We control the full stack, giving us the freedom to develop new approaches across models, agent harnesses, and search infrastructure to use search more effectively, design tools and execution environments around learned behavior, and improve retrieval to support how agents actually work.

Anthropic

Anthropic is building AI systems that are safe, beneficial, and transformative, with the mission to develop AI that benefits humanity. This role focuses on applying Claude to scientists and engineers working on hard problems in energy, materials, hardware design, and other physical-world fields. The team builds the necessary software, including agent harnesses, integrations, evaluations, and working processes, to turn a partner's problem into something Claude can make progress on and validate. This is an engineering role where you will work closely with domain experts, become proficient in using Claude for their specific fields, and prototype quickly. The primary focus is on building and hardening systems to make this work repeatable, rather than just configuring existing products. A science or engineering background is beneficial, but the ability to earn the trust of expert researchers, translate fuzzy problems into concrete and verifiable tasks, and ship solutions is paramount.

$405k - $485k

Sourcegraph is building AI tools to address challenges in the software industry, especially as codebases grow and agentic development becomes prevalent. The Code Understanding team focuses on areas like Deep Search, Query Assist, and Smart Hovers, which are powered by agent engineering. This role is for a staff-level technical leader responsible for the direction of models, evaluations, and agentic systems within the Code Understanding team. The goal is to improve product performance, speed, and cost-effectiveness by integrating deterministic systems with AI. This position offers significant agency over technical direction and a direct impact on product development.

Anthropic

The People Products team at Anthropic is dedicated to shaping the future of AI at work, focusing on critical aspects of the employee lifecycle. This role involves defining the blueprint for AI in professional environments, from hiring and manager effectiveness to onboarding and fostering company culture through innovative tools. You will work directly with Anthropic's advanced AI, Claude, tackling complex, unsolved problems and moving rapidly from prototype to production. This is an opportunity for individuals who are passionate about improving people practices and want to make a significant impact at a leading AI company.

$405k - $485k

Taktile is seeking a Data Engineer (AI Enablement) to join their rapidly growing team, empowering financial institutions to become AI-native. As the first dedicated Data Engineer for AI Enablement, you will be responsible for building internal tools, agents, and integrations to enhance team efficiency. This role involves end-to-end project ownership, collaborating with leads across Engineering, GTM, and Ops to identify and scale high-impact workflows. You will embed across various departments to pinpoint areas where AI can reduce operational friction, drive adoption through workshops and support, and partner with Security and IT to ensure data protection.

Unusual Ventures is seeking an AI Engineer to join their team. This role involves building AI-powered agents and pipelines to enhance talent intelligence, sourcing, and research for their portfolio companies. You will also develop tools for diligence support, firm knowledge management, operations automation, and portfolio support, directly impacting the firm's usefulness to the founders they back. The position emphasizes building practical solutions that address real user problems and iterating quickly.

The Agentic Interfaces team at Datadog is seeking a Staff Applied Scientist to lead the evaluation strategy for AI agent integrations. This role focuses on defining and building the measurement systems that ensure AI agents interacting with Datadog's observability, security, and software delivery data are continuously improving. You will be responsible for establishing metrics that cover answer quality, tool selection accuracy, retrieval relevance, latency, and cost, and will partner with AI engineers to implement these systems. This is an opportunity to tackle open research questions in agent-data interaction, such as evaluating non-deterministic trajectories, scoring tool selection in large catalogs, and building scalable measurement systems that benefit both first-party and third-party agents.

$276k - $345k

Asana's AI Teammates are autonomous agents designed to integrate into enterprise workflows, proactively driving work forward. The AI Teammates Platform (AITP) team is responsible for the execution engine and platform layer that ensures these agents are effective, reliable, and scalable. This involves building core systems for agentic execution, including model evaluation, rollout strategies, infrastructure for detecting blocked work, quality assurance, tool orchestration, and platform APIs that power AI across Asana's engineering efforts. The team focuses on creating a composable foundation rather than a monolithic feature, enabling AI Teammates to reason, act, and autonomously improve.

$248k - $282k