AI Inference Platform Engineer
$200k - $250k • Chicago • Full-time
Posted 1mo ago
Job Location
Chicago
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
We are seeking an AI Inference Platform Engineer to construct, manage, and enhance the systems responsible for serving large language, vision, multimodal, and embedding models across DRW. This role serves as the firm's primary interface for modern AI models, guiding them from initial evaluation to reliable production deployment. You will collaborate on inference runtimes, distributed systems, and production platform engineering, requiring a strong understanding of GPUs. Your responsibilities will include end-to-end ownership of the serving platform, onboarding new models, assessing quality and performance across different serving configurations, scheduling workloads for various tenants, and continuously improving the inference fleet's latency, throughput, utilization, reliability, and cost-efficiency.
Responsibilities
- Optimize LLM inference performance on NVIDIA GPUs and inference runtimes.
- Build comprehensive performance profiling and observability systems to identify bottlenecks from GPU kernels to multi-node inference.
- Design and optimize KV cache and distributed inference architectures, including caching, routing, memory tiering, and prefill/decode strategies.
- Manage day-0 model onboarding, selecting appropriate runtimes, precision, sharding, memory, batching, cache policies, and serving configurations.
- Maintain validated performance profiles for models and hardware, including regression testing for performance and quality.
- Measure and monitor quality equivalence across serving configurations, ensuring in-house serving matches reference model quality.
- Manage the production serving lifecycle of models, including versioning, compatibility, staging, canarying, promotion, rollback, and retirement.
- Automate model deployment, distribution, production readiness, observability, and reliable operation in partnership with SRE and platform teams.
- Optimize model placement, scaling, and resource allocation to improve utilization and cost efficiency while meeting performance and reliability requirements.
- Design and operate multi-tenant scheduling and isolation for shared GPU capacity, balancing SLOs, throughput, and priority.
Requirements
- Hands-on experience serving LLMs on NVIDIA GPUs, with familiarity across current and emerging architectures and compute/memory bottlenecks.
- Deep expertise in at least one modern inference runtime (e.g., TensorRT-LLM, vLLM, SGLang).
- Practical knowledge of inference optimization techniques (e.g., continuous batching, speculative decoding, quantization, paged attention).
- Understanding of KV cache architecture and optimization.
- Experience measuring model quality equivalence across serving configurations.
- Experience designing and tuning distributed inference systems (e.g., tensor parallelism, multi-node deployments).
- Experience with multi-tenant GPU scheduling and workload isolation.
- Proficiency with GPU performance and observability tooling (e.g., Nsight, DCGM, Prometheus, Grafana).
- Strong Linux and systems performance fundamentals.
- Production experience with model serving infrastructure, including CI/CD, automated testing, and observability.
- Measurement-driven approach to performance optimization.
- Ability to take ownership of performance problems across various boundaries.
- Ability to move quickly and reprioritize while maintaining high standards for production systems.
- Understanding of reliability, predictability, and performance in AI systems for trading workflows.
- Ability to evaluate unfamiliar models, runtimes, and hardware quickly and make sound engineering decisions.
- Clear communication skills for explaining complex performance tradeoffs.
Benefits
- Annual base salary range: $200,000 to $250,000
