Senior Backend Engineer, Inference Platform
$200k - $250k • Remote • San Francisco, CA, New York, NY (Remote)
Posted 4d ago
Job Location
San Francisco, CA, New York, NY (Remote)
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
You.com is building the AI Search Infrastructure that powers modern AI systems, aiming to create a trusted knowledge layer for agents, applications, and enterprises. Our platform combines proprietary vertical indexes with LLM-optimized retrieval systems. We are seeking a Senior Backend Engineer to own and scale the inference gateway, which is the critical layer that handles all model calls. This role involves routing, fallback, caching, metering, and enforcement within strict latency budgets, sitting between model serving and product development.
Responsibilities
- Own and scale the inference gateway, including provider and model selection, fallback, and caps.
- Build an intelligence layer to autonomously select the optimal model for each turn in a task.
- Implement routing decisions considering prefix-cache behavior, batching, and time to first token, alongside price and availability.
- Develop cost attribution and metering across providers, including cached-token accounting for billing and enforcement.
- Build request-path enforcement mechanisms that degrade gracefully without interrupting customer inference calls.
- Ship inference-time changes in shadow mode with supporting measurements to validate their effectiveness.
- Gain experience at an inference provider or GPU cloud, or contribute to an open-source serving stack.
- Fine-tune and evaluate models in production.
- Implement usage metering, billing, or cost optimization at scale.
Requirements
- Hands-on production experience with model serving on stacks like vLLM or SGLang.
- Working understanding of continuous batching, KV and prefix caching, and the trade-offs between time to first token and throughput.
- Experience tuning a serving stack against live traffic and understanding practical p99 performance drivers.
- Fluency with provider APIs, including tokenization, prompt-caching semantics, and differences in usage/cached token reporting and pricing.
- Experience building or operating a proxy, gateway, or data plane on the critical path of production traffic, with strong instincts for failing open.
- Comfort with streaming responses, including handling partial state, mid-stream failures, and post-stream operations.
- Strong backend generalist skills (Python, Kubernetes), with the ability to learn new technologies.
- Emphasis on trajectory and recent growth over years of experience.
Benefits
- Hubs in San Francisco and New York City offering regular in-person gatherings and co-working sessions
- Flexible PTO with U.S. holidays observed and an end of year shutdown in December
- Competitive health insurance plan covering 100% of the policyholder and 75% for dependents
- Function Health membership
- 12 weeks of paid parental leave in the US
- 401k program with a 3% match, vested immediately
- $500 work-from-home stipend
- $600 technology stipend for cell phone and internet expenses
- $1,200 per year Health & Wellness Allowance
- Opportunity to collaborate with a team at the forefront of AI research