LLM Inference Frameworks and Optimization Engineer
$160k - $230k • Remote • San Francisco, Singapore, Amsterdam
Posted 1mo ago
Job Location
San Francisco, Singapore, Amsterdam
Tech Stack
Remote Work Policy
Fully remote
Categories
LLM Engineer
About the job
Together.ai is building state-of-the-art infrastructure for efficient and scalable inference of large language models (LLMs). The company's mission is to optimize inference frameworks, algorithms, and infrastructure to push the boundaries of performance, scalability, and cost-efficiency. They are seeking an Inference Frameworks and Optimization Engineer to design, develop, and optimize distributed inference engines for multimodal and language models at scale. This role will focus on low-latency, high-throughput inference, GPU/accelerator optimizations, and software-hardware co-design, ensuring efficient large-scale deployment of LLMs and vision models. This position offers a unique opportunity to shape the future of LLM inference infrastructure and ensure scalable, high-performance AI deployment across diverse applications.
Responsibilities
- Design and develop fault-tolerant, high-concurrency distributed inference engines for text, image, and multimodal generation models.
- Implement and optimize distributed inference strategies like Mixture of Experts (MoE) parallelism, tensor parallelism, and pipeline parallelism for high-performance serving.
- Apply CUDA graph optimizations, TensorRT/TRT-LLM graph optimizations, PyTorch compilation (torch.compile), and speculative decoding to enhance efficiency and scalability.
- Collaborate with hardware teams on performance bottleneck analysis and co-optimize inference performance for GPUs, TPUs, or custom accelerators.
- Work closely with AI researchers and infrastructure engineers to develop efficient model execution plans and optimize end-to-end model serving pipelines.
Requirements
- 3+ years of experience in deep learning inference frameworks, distributed systems, or high-performance computing.
- Familiarity with at least one LLM inference framework (e.g., TensorRT-LLM, vLLM, SGLang, TGI).
- Background knowledge and experience in GPU programming (CUDA/Triton/TensorRT), compiler, model quantization, or GPU cluster scheduling.
- Deep understanding of KV cache systems like Mooncake, PagedAttention, or custom variants.
- Proficiency in Python and C++/CUDA for high-performance deep learning inference.
- Deep understanding of Transformer architectures and LLM/VLM/Diffusion model optimization.
- Knowledge of inference optimization techniques such as workload scheduling, CUDA graph, compiled kernels, and efficient kernels.
- Strong analytical problem-solving skills with a performance-driven mindset.
- Excellent collaboration and communication skills across teams.
- Experience in developing software systems for large-scale data center networks with RDMA/RoCE (nice-to-have).
- Familiarity with distributed filesystems (e.g., 3FS, HDFS, Ceph) (nice-to-have).
- Familiarity with open-source distributed scheduling/orchestration frameworks like Kubernetes (K8S) (nice-to-have).
- Contributions to open-source deep learning inference projects (nice-to-have).
Benefits
- Competitive compensation
- Startup equity
- Health insurance
- Other competitive benefits