Senior Staff LLM Inference Engineer
$195k - $285k • Santa Clara • FullTime
Posted 2mo ago
Job Location
Santa Clara
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
LLM Engineer
About the job
D-Matrix Frontier Group is at the leading edge of LLM inference on heterogeneous hardware, working across the full stack from use case identification and kernel optimization to building proof-of-concept systems. We build the tools, runtimes, and frameworks that enable frontier AI models to run efficiently and cost-effectively on diverse hardware, including D-Matrix silicon, CPUs, GPUs, and custom accelerators. This applied research and engineering team moves fast, ships real systems, and collaborates directly with product and hardware teams to shape the roadmap. We are seeking end-to-end inference engineers who are comfortable taking a novel research idea to a deployed, optimized system, working at every layer of the inference stack from kernel-level optimization to distributed orchestration and high-level serving APIs.
Responsibilities
- Identify and prototype emerging LLM inference use cases for heterogeneous hardware.
- Build proof-of-concept systems showcasing D-Matrix capabilities.
- Develop and tune custom kernels and operator-level optimizations for maximum throughput and minimum latency.
- Drive quantization, sparsity, and batching strategies for the D-Matrix computational model.
- Build and maintain inference runtimes, serving frameworks, and evaluation tooling.
- Contribute to distributed inference systems, including tensor/pipeline parallelism and disaggregated prefill/decode.
- Provide inference workload insights to firmware and compiler teams.
- Partner with product and business development to translate POCs into customer-facing demonstrations.
- Contribute to technical publications, whitepapers, and open-source projects.
Requirements
- Bachelor's degree in Computer Science, Electrical Engineering, or related field, with 10+ years of relevant engineering experience, or equivalent.
- Master's or PhD in Computer Science, Electrical Engineering, or related field preferred, with 6+ years of relevant industry experience.
- Strong proficiency in Python and C/C++.
- Hands-on experience optimizing LLM inference, including attention kernels, KV cache, batching strategies, and quantization (INT8/FP8/INT4).
- Experience with at least one major inference framework (vLLM, SGLang, TensorRT-LLM, ONNX Runtime, or similar) at a contributor level.
- Familiarity with GPU kernel programming (CUDA/Triton) and performance profiling tools.
- Experience with heterogeneous compute deployments and scheduling inference workloads across dissimilar hardware.
- Familiarity with custom silicon or ASIC-based inference.
- Experience with distributed inference (tensor parallelism, pipeline parallelism, disaggregated serving).
- Contributions to open-source inference or ML systems projects.
- Experience with production inference serving at scale.
- Familiarity with speculative decoding, mixture-of-experts routing, or long-context-serving techniques.
- Working familiarity with the JAX Scaling Book or equivalent systems-level understanding of modern LLM training and inference.
Benefits
- Competitive compensation
- Equity
- Benefits