Staff Software Engineer, Inference Infrastructure
Remote • San Francisco • FullTime
Posted 6mo ago
About the job
Cohere is seeking Members of Technical Staff to join the Model Serving team. This role focuses on developing, deploying, and operating the AI platform that delivers Cohere's large language models via API endpoints. You will work closely with various teams to deploy optimized NLP models into production environments, ensuring low latency, high throughput, and high availability. The position also offers the opportunity to interact with customers and build customized deployments to meet their specific needs.
Responsibilities
- Develop, deploy, and operate the AI platform for serving large language models.
- Deploy optimized NLP models to production in low latency, high throughput, and high availability environments.
- Interface with customers to create customized deployments.
- Build mission-critical systems and ensure smooth operations.
- Troubleshoot complex technical challenges in evolving environments.
Requirements
- 5+ years of engineering experience running production infrastructure at scale.
- Experience designing large, highly available distributed systems with Kubernetes and GPU workloads.
- Experience with Kubernetes development, production coding, and support.
- Experience with multi-cloud environments (GCP, Azure, AWS, OCI) and on-prem/hybrid serving.
- Experience designing, deploying, supporting, and troubleshooting complex Linux-based computing environments.
- Experience in compute/storage/network resource and cost management.
- Excellent collaboration and troubleshooting skills.
- Grit and adaptability to solve complex technical challenges.
- Familiarity with computational characteristics of accelerators (GPUs, TPUs, custom) and their impact on inference latency and throughput.
- Strong understanding or working experience with distributed systems.
- Experience in Golang, C++, or other high-performance server languages.
Benefits
- Weekly lunch stipend ($75/£75 or local equivalent).
- Full health and dental benefits with a mental health budget.
- RRSP matching, 401K, Pension Scheme.
- 100% Parental Leave top-up for up to 6 months.
- Annual enrichment benefits (arts & culture, fitness/wellness, quality time, workspace improvement).
- Education & learning stipend (conferences, courses, coaching).
- 6 weeks of paid vacation (30 working days).
- Budget for traveling to other offices (for remote employees).
- Annual company offsite.
- $500 home office stipend.