New: upload your resume and get matched to AI jobs. Free to start →

Principal AI Software Engineer

$143k - $275k • United States, Washington, Redmond; United States, California, Mountain View; United States, Oregon, Hillsboro • Full-time

Posted 9d ago

Job Location

United States, Washington, Redmond; United States, California, Mountain View; United States, Oregon, Hillsboro

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

LLM Engineer

About the job

The Compute System Architecture (CSA) team within the System Planning and Architecture (SPARC) organization is seeking a Principal AI Software Engineer. This role is at the forefront of systems architecture and technology pathfinding for Azure's hardware systems and infrastructure. You will be responsible for defining and evaluating novel systems architecture innovations through hardware/software co-design, advancing them through technical readiness for productization, and influencing hardware architecture and industry alignment for future offerings. This is a unique opportunity to combine technical capabilities with cross-team collaboration and business insight to drive innovation for Microsoft's expanding cloud business.

Responsibilities

  • Lead full system software prototyping to develop proof-of-concepts for evaluating hardware/software co-designed capabilities for memory TCO reduction.
  • Develop deep insights through workload characterization to identify systems optimization opportunities.
  • Collaborate with workload experts to engineer TCO-optimized solutions for Azure compute fleets.
  • Influence hardware architecture and industry alignment with data-driven analysis.
  • Lead characterization and optimization of LLM inference workloads focusing on KV Cache across various memory tiers.
  • Develop proof-of-concepts and evaluation frameworks for memory-tiering architectures for AI inference.
  • Design and execute workload characterization studies for AI workloads to quantify performance metrics.
  • Analyze end-to-end data movement across subsystems to identify optimization opportunities.
  • Develop software prototypes and instrumentation to evaluate KV Cache offload and memory-overcommit techniques.
  • Build performance models and simulation frameworks to predict the impact of memory hierarchy innovations.

Requirements

  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
  • 12+ years of experience in systems software (OS kernel, memory management, I/O stacks, Virtualization) with a demonstrated track record of success.
  • 10+ years of experience leading significant hardware/software co-design projects involving CPU and/or systems architecture.
  • Deep expertise in Linux kernel internals, memory management, I/O subsystems, NUMA, DMA, and GPU/CPU/storage/network data paths.
  • Hands-on experience with NVIDIA GPU software stacks including CUDA, NCCL, GPUDirect Storage (GDS), and GPUDirect RDMA (GDR).
  • Understanding of AI inference infrastructure, large GPU clusters, inference serving architectures, and workload performance optimization.
  • Experience characterizing and optimizing KV Cache intensive workloads.
  • Experience with inference frameworks such as vLLM, SGLang, and TensorRT-LLM.
  • Familiarity with KV Cache technologies including LMCache, SGLang HiCache, cache offload, cache sharing, and memory tiering approaches.
  • Experience designing or extending inference runtimes, scheduling systems, memory management components, or KV Cache subsystems.
  • Understanding of disaggregated prefill/decode architectures, distributed inference serving, and multi-node cache-sharing topologies.
  • Experience with CXL memory expansion, memory pooling, and memory tiering solutions.
  • Software development skills in C/C++, Python, CUDA, and distributed systems.
  • Skilled in partnering and influencing architects, hardware engineers, and software leads.
  • Ability to manage through ambiguity and bring clarity and results orientation.