Senior AI Infrastructure Engineer, Model Serving Platform
$216k - $270k • Remote • San Francisco, CA; New York, NY
Posted 2mo ago
Job Location
San Francisco, CA; New York, NY
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
As a Software Engineer on the ML Infrastructure team, you will design and build platforms for scalable, reliable, and efficient serving of LLMs. Our platform powers cutting-edge research and production systems, supporting both internal and external use cases across various environments. You'll work in a highly collaborative environment, bridging research and engineering to deliver seamless experiences to our customers and accelerate innovation across the company.
Responsibilities
- Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.
- Build an internal platform to empower LLM capability discovery.
- Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
- Conduct architecture and design reviews to uphold best practices in system design and scalability.
- Develop monitoring and observability solutions to ensure system health and performance.
- Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment.
Requirements
- 5+ years of experience building large-scale, high-performance backend systems.
- Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++).
- Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.).
- Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.
- Experience with containers and orchestration tools (e.g., Docker, Kubernetes).
- Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
- Proven ability to solve complex problems and work independently in fast-moving environments.
Benefits
- Base salary, equity, and benefits.
- Comprehensive health, dental and vision coverage.
- Retirement benefits.
- A learning and development stipend.
- Generous PTO.
- Commuter stipend.