Senior Staff Inference Performance Engineer
$175k - $265k • Remote • Santa Clara • FullTime
Posted 2d ago
About the job
d-Matrix is seeking a Senior Staff Inference Performance Engineer to join their Inference Cloud Engineering team. This role is responsible for the end-to-end performance of workloads deployed on d-Matrix's inference services, ensuring they perform at least as well as on bare metal. The engineer will diagnose and isolate performance issues across the full serving path, from orchestration to inference internals, and will be responsible for either fixing problems directly or handing them off to the appropriate owning team with clear evidence. The role involves assessing scalability, defining node requirements for optimal deployment, and building telemetry and benchmarks to support these efforts. The ideal candidate will mentor other engineers in a profiling-first methodology.
Responsibilities
- Diagnose performance problems across the full serving path, including orchestration and inference internals.
- Isolate performance issues to a specific layer and owner, and decide on the best course of action (fix, route, or both).
- Write clear, reproducible problem statements for cross-stack findings.
- Ensure throughput, latency, and stability targets for orchestrated workloads are met by identifying and closing performance gaps.
- Assess scalability requirements for fleet operations, including routing, placement, and resource contention.
- Define and capture workload node requirements in the orchestration layer's configuration for performant deployment.
- Build lightweight telemetry and benchmarks to isolate or hand off problems in prototype stacks.
- Develop playbooks for recurring investigations and mentor engineers on the profiling-first method.
Requirements
- 8+ years of hands-on performance or systems engineering experience, including resolving complex regressions across multiple stack layers.
- Proven ability to find performance overhead through profiling in unfamiliar, sparsely documented systems.
- Deep systems fundamentals, including Linux internals, NUMA, CPU pinning, PCIe, memory/cache behavior, and network paths.
- Experience running performance-sensitive workloads on Kubernetes, including identifying and addressing performance bottlenecks.
- Working knowledge of LLM inference serving concepts (batching, KV-cache, prefill/decode, quantization, speculative decoding).
- Proficiency in Python and C/C++ for instrumentation, debugging, and patching.
- Clear technical writing skills for cross-functional teams.
- Sound judgment regarding problem ownership and effective handoff of issues.
- Fluency with AI and agentic tools for investigation, code search, log triage, and drafting instrumentation, with verification discipline.