Staff Applied AI Inference Engineer
$215k - $260k • Remote • San Francisco, CA - US • FullTime
Posted 1mo ago
About the job
Crusoe is seeking a Staff Applied AI Inference Engineer to accelerate the abundance of energy and intelligence by optimizing large language models for production environments. This role involves owning the inference stack end-to-end, focusing on profiling, applying modern optimization techniques, and deep dives into serving code to enhance speed, reduce cost, and improve reliability. The work is hands-on, involving coding, profiling, and low-level optimization, with a customer-facing component to tailor deployments to specific needs and ensure performance gains are realized in real-world scenarios. You will transition workloads from proof-of-concept to fully monitored production services, making this a core systems and performance engineering role on demanding AI workloads.
Responsibilities
- Implement and refine current inference techniques in production.
- Design and optimize serving architectures, including prefill and decode disaggregation and request routing.
- Profile and analyze performance down to the kernel level within serving frameworks like vLLM and SGLang to identify and resolve performance issues.
- Adapt and scale optimization methods across various ML models, with a focus on large language models.
- Profile and tune deployments for latency, throughput, and cost, ensuring dependability under real traffic.
- Collaborate with customer engineering teams to tailor deployments to their specific models and constraints.
- Build and support software and product features for the inference stack in production, primarily using Python.
- Rapidly experiment with fuzzy goals, shape them into clear specifications and proofs of concept, and ship well-tested results.
- Own delivery end-to-end, from initial experiments to production deployment, focusing on performance goals and product requirements.
- Navigate ambiguity and make informed decisions on tradeoffs and tooling.
- Take ownership and accountability for work, and foster the same in teammates.
Requirements
- Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, Mathematics, or a related field.
- Hands-on experience shipping production code in general-purpose languages like Python or C++, with a strong preference for Python.
- Familiarity with methods for optimizing LLMs for high throughput and low latency inference.
- Experience with modern LLM serving frameworks such as vLLM or SGLang.
- Proficiency in profiling and analyzing performance down to the kernel level.
- Solid understanding of GPU architecture and behavior.
- Demonstrated interest and hands-on experience with large language models.
- Working knowledge of AI/ML pipelines and the end-to-end ML model development and deployment lifecycle.
- Strong communication skills, especially in explaining complex technical topics to customers and teammates.
- Experience making software systems faster, particularly for large language models.
- Experience with CUDA or comparable technologies.
- Strong software engineering fundamentals and a record of building and shipping AI/ML inference systems.
- Experience with Docker and Kubernetes.
- Prior experience building or tuning AI/ML projects, especially in a customer-facing role.
Benefits
- Competitive compensation and equity packages
- Restricted Stock Units
- Paid time off, paid holidays & leave of absence programs
- Comprehensive health, dental & vision insurance
- Employer contributions to HSA account
- Paid parental leave
- Paid life insurance, short-term and long-term disability
- Professional development & tuition reimbursement
- Mental health & wellness support
- Commuter benefits (parking & transit)
- Cell phone stipend