Forward Deployed Engineer, Inference
$240k - $415k • Remote • United States • FullTime
Posted 16h ago
About the job
Cohere is seeking a Staff Forward Deployed Engineer to bridge the gap between frontier AI capabilities and complex customer realities. This role involves architecting high-performance, resilient machine learning systems within enterprise environments, acting as the primary technical liaison between Cohere's core engineering teams and strategic partners. You will own the architecture for customer-specific inference challenges, ensuring our large language models operate reliably, securely, and at peak performance within client infrastructure. The goal is to achieve exceptionally low latency, high throughput, and seamless resilience in production deployments.
Responsibilities
- Embed with customer engineering teams to architect the integration of AI models into their production environments.
- Design advanced model serving topologies, including multi-tenant architectures, intelligent model routing, and cross-datacenter failover strategies.
- Optimize 'AI-in-a-box' and private deployments, balancing Kubernetes scale with model performance.
- Benchmark and profile customer LLM workloads to identify and resolve performance degradations.
- Accelerate hardware validation by supporting Cohere models on next-generation accelerators.
- Translate field challenges into platform upgrades and contribute feedback to core product features.
- Ensure Day-1 production reliability by assisting customers with deployment patterns, SLOs, and observability.
Requirements
- 8+ years of experience in production infrastructure, large-scale distributed systems, or enterprise software deployment.
- Deep technical expertise in production Kubernetes, including multi-cluster and hybrid/multi-cloud environments (GCP, Azure, AWS, OCI).
- Strong understanding of LLM architecture and inference dynamics.
- Hands-on experience with modern inference engines and optimization stacks (e.g., vLLM, TensorRT-LLM).
- Familiarity with distributed serving techniques like prefill/decode disaggregation and multi-GPU workloads.
- Exceptional customer-facing technical leadership and problem-scoping abilities.
- High agency, grit, and adaptability for rapid production code deployment and client management.
- Strong programming skills in Python, Go, or C++.
Benefits
- Weekly lunch stipend ($75/£75 or equivalent).
- Full health and dental benefits with a mental health budget.
- Retirement savings plans (RRSP matching, 401K, Pension Scheme).
- 100% Parental Leave top-up for up to 6 months.
- Annual enrichment benefits (arts, culture, fitness, workspace improvement).
- Education and learning stipend for conferences, courses, and coaching.
- 6 weeks (30 working days) of paid vacation.
- Budget for travel to other offices (for remote employees).
- Annual company offsite.
- Co-working benefit for remote employees not near an office.
- $500 home office stipend.