AI Product Engineer - ClickStack
£90k - £135k • United Kingdom • Full-time
Posted 25d ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
ClickStack is seeking an AI Product Engineer to develop agentic capabilities for their open-source observability platform. This role focuses on building AI systems that can investigate incidents, propose root causes, and provide concise summaries for on-call engineers. You will be responsible for the end-to-end agent stack, including context engineering, tool design, evaluations, tracing, and cost management, ensuring the agents function effectively in production. The position also involves making ClickStack a robust platform for running AI workloads by developing MCP servers, SDKs, and integrations, and collaborating with the open-source community.
Responsibilities
- Build agents that investigate incidents, surface anomalies, and determine root causes using ClickStack as their foundation.
- Develop a library of reusable skills that capture team debugging and incident response processes for agent utilization.
- Own the agent stack from context engineering and tool design to evals, tracing, and cost, ensuring production readiness.
- Create MCP servers, SDKs, and integrations to enable customer agents to interact with telemetry and take actions.
- Collaborate with open-source contributors and customers, debug issues, and integrate learnings back into the product.
- Address challenges such as latency, cost, context window limits, eval coverage, and hallucinations in production telemetry.
Requirements
- Proven experience building agents with opinions on context engineering, tool design, skill vs. tool usage, and framework limitations.
- Production-oriented mindset focusing on p99 latency, cost per task, and system maintainability.
- Ability to move quickly, ship frequently, and learn from failures.
- Strong understanding of developer tools and excellent developer experience (DX).
- Comfort with ambiguity and a strong sense of ownership.
- 5+ years of software engineering experience, with 1-2 years on LLM-powered systems or agents in production.
- Strong backend skills in TypeScript/Node.js and/or Python.
- Hands-on experience building agents with multi-step tool use, planning, memory, and error recovery.
- Experience designing skills (e.g., Markdown-based workflow encodings) and understanding when to use skills vs. tools.
- Experience with MCP: building servers, designing tools, and considering auth, scoping, and observability for agentic systems.
- Proficiency in evaluation practices like golden sets, LLM-as-judge, and regression detection.
- SQL proficiency, including writing ClickHouse queries.
- Comfort with Docker and Kubernetes.
- Active participation in open source and the developer community.
Benefits
- Flexible work environment
- Employer contributions towards healthcare
- Company stock options
- Flexible time off (US) / generous entitlement (other countries)
- USD$500 Home office setup for remote employees
- Global Gatherings and company-wide offsites