Senior Software Engineer, LLM Platform
$176k - $220k • Remote • San Francisco, CA • FullTime
Posted 3mo ago
Job Location
San Francisco, CA
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
LLM Engineer
About the job
Handshake is seeking a Senior LLM Platform Engineer to join their Data and ML Platform team. This role is crucial for building and operating the shared LLM control plane, including LiteLLM gateways, provider integrations, access controls, observability, and cost management. You will also contribute to adjacent platform systems for workflow orchestration, model serving, and developer enablement, collaborating closely with various engineering and data science teams. This is an infrastructure-heavy role focused on ensuring the reliability and scalability of LLM services within Handshake's core career marketplace and Handshake AI initiatives.
Responsibilities
- Build and operate LiteLLM-based AI gateways and shared LLM clients.
- Manage provider and model onboarding, routing, failover, rate limits, and capacity planning.
- Develop self-service workflows for virtual-key, model-access, budget, and credential management.
- Establish SLOs, observability, cost attribution, and alerts for LLM traffic.
- Qualify and roll out new models, providers, SDKs, and gateway configurations.
- Support hosted and self-hosted inference through a consistent platform interface.
- Partner with product, AI, and FDE teams to translate recurring delivery problems into platform capabilities.
- Contribute to broader Data and ML Platform systems, including workflow orchestration and model serving.
- Participate in team on-call and support, improving runbooks, automation, and reliability.
Requirements
- Strong production software engineering experience in Python, TypeScript, Go, or similar.
- Experience building or operating high-throughput API gateways, proxies, or multi-tenant platform services.
- Hands-on experience with Kubernetes, Terraform, CI/CD, and production service ownership.
- Practical experience with LLM provider APIs, including streaming, long-running requests, retries, timeouts, cancellation, and rate limits.
- Experience with authentication, quotas, credential management, tenant isolation, and auditability.
- Experience building observability for distributed systems and leading production incident response.
- Experience with usage metering, cost attribution, capacity planning, or FinOps.
- Experience with data or ML platform systems like BigQuery, Airflow, streaming pipelines, model serving, or ML observability.
- Strong judgment in ambiguous environments and a bias toward simple, reliable systems that scale.
Benefits
- Equity in a fast-growing company
- 401(k) match
- Competitive compensation
- Financial coaching
- Paid parental leave
- Fertility benefits
- Parental coaching
- Medical, dental, and vision insurance
- Mental health support
- $500 wellness stipend
- $2,000 learning stipend
- Ongoing development opportunities
- Internet stipend for remote work
- Commuting stipend
- Free lunch/gym in SF office
- Flexible PTO
- 15 holidays + 2 flex days
- Team outings
- Referral bonuses