Distributed LLM Inference Engineer
$170k - $245k • San Francisco • FullTime
Posted 4mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
LLM Engineer
About the job
Anyscale is seeking a Distributed LLM Inference Engineer to develop systems and optimizations that push the boundaries of performance for large-scale inference. This critical role will contribute to Anyscale's market-leading position in AI infrastructure. You will work closely with product teams to deliver end-to-end solutions for batch and online inference, integrating Ray Data and LLM engines to achieve cost-effective, large-scale ML inference. You will also integrate with open-source software like vLLM, contribute to Anyscale's solutions, and improve open-source projects. Staying abreast of the latest advancements in open-source and research is key to implementing and extending best practices.
Responsibilities
- Ship end-to-end solutions for batch and online inference at high scale.
- Integrate Ray Data and LLM engines for optimized, low-cost large-scale ML inference.
- Integrate with open-source software like vLLM and contribute improvements.
- Follow and implement state-of-the-art techniques from open-source and research communities.
Requirements
- Familiarity with running ML inference at large scale with high throughput and low latency.
- Familiarity with deep learning and deep learning frameworks (e.g. PyTorch).
- Solid understanding of distributed systems and ML inference challenges.
Benefits
- Stock Options
- Healthcare plans, with premiums covered by Anyscale at 99% for both employees and dependents
- 401k Retirement Plan
- Education & Wellbeing Stipend
- Paid Parental Leave
- Fertility Benefits
- Paid Time Off
- Commute reimbursement
- 100% of in-office meals covered