Model Behavior Architect- Function Calling
Remote • London • Full-time
Posted 2mo ago
Job Location
London
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full-time
Categories
LLM Engineer
About the job
As a Model Behavior Architect on the Function Calling team, you will be at the forefront of defining and measuring how Large Language Models (LLMs) utilize tools, invoke functions, and orchestrate complex agentic workflows. We are seeking individuals with a strong background in engineering, machine learning, and LLMs, who possess expertise in model evaluation, policy writing, and creating evaluation pipelines specifically for tool use and function calling. Your primary responsibility will be to collaborate closely with our Science team to establish benchmarks for function calling performance, covering aspects such as accurate parameter selection, schema adherence, multi-step tool orchestration, error recovery, and agentic reasoning. This role is ideal for those passionate about tackling cutting-edge, open-ended research challenges and translating insights into best-in-class models.
Responsibilities
- Interact with models to identify areas for improvement in function calling and tool use behavior.
- Gather internal and external feedback on tool-calling behavior to define improvement scopes.
- Design and implement evaluations, data guidelines, data generation processes, and synthetic tool environments and APIs.
- Identify and resolve edge case behaviors, including malformed arguments, hallucinated functions, and incorrect tool selection, through rigorous testing.
- Develop robust evaluation pipelines for the function-calling capabilities of model candidates.
- Collaborate effectively with AI Scientists.
Requirements
- Deep understanding of API design, structured outputs, and schema specification (e.g., JSON Schema).
- Deep understanding of engineering and code behavior.
- Deep understanding of LLM agents, including reasoning, planning, and multi-step tool use.
- Prior knowledge in training and optimizing model behavior.
- Expertise in building robust evaluations.
- Proven track record of delivering innovative, out-of-the-box solutions to address real-world constraints.