New: upload your resume and get matched to AI jobs. Free to start →

Senior AI Developer

$200k - $250k • United States • Full-time

Posted 14d ago

Remote Work Policy

On-site

Employment Type

Full-time

Visa Sponsorship

The posting says they sponsor visas

Categories

AI Engineer

About the job

ClickUp is building the future of work with an AI-native converged workspace that unifies tasks, docs, chat, calendar, and enterprise search. We are seeking a Senior AI Engineer to focus on AI evaluations, owning the systems that measure and improve the quality of ClickUp's AI. This role involves building evaluation systems, core agent harnesses, and backend systems to ensure AI-powered capabilities are scalable, reliable, and secure, while directly addressing user problems with LLMs.

Responsibilities

  • Design, build, and own the AI evaluation system, including golden datasets, LLM-as-judge, automated scorers, regression detection, and quality tracking.
  • Build and evolve the core agent harness and AI for evaluations, partnering with product and engineering.
  • Develop self-improvement loops where agents modify other agents or prompts, rerun evaluations, and iterate automatically.
  • Benchmark new models, manage model selection, routing, and fallback strategies across providers, balancing quality, latency, and cost.
  • Architect, design, and implement scalable AI platform services for LLM deployment, orchestration, and lifecycle management.
  • Build and maintain robust APIs and backend systems to integrate AI features into the core platform.
  • Develop infrastructure for model serving, monitoring, logging, and automated evaluation for high reliability and performance.
  • Drive best practices in AI privacy, security, and compliance, including data anonymization and secure data handling.
  • Collaborate cross-functionally with product, frontend, and data teams to deliver user-centric AI experiences.

Requirements

  • 5+ years of software engineering experience, with 2+ years building production LLM or ML systems.
  • Hands-on experience building LLM evaluation systems in production, tied to real product quality.
  • Strong experiment design and statistical rigor, including A/B testing and metric analysis.
  • Proven track record of applying LLMs and AI models to customer-facing product features.
  • Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs.
  • Deep expertise in backend engineering, distributed systems, and cloud-native technologies (e.g., Kubernetes, Docker, AWS/GCP/Azure).
  • Strong programming skills in Python (preferred), Go, TypeScript, or similar backend/AI platform languages.
  • Proficiency in orchestration frameworks and workflow engines (e.g., LangGraph, Airflow, Ray).
  • Familiarity with search and retrieval technologies and their integration into AI-driven applications.
  • Excellent collaboration and communication skills, with a track record of working effectively in cross-functional teams.