New: upload your resume and get matched to AI jobs. Free to start →

Principal Software Engineering, AI Infrastructure

$143k - $275k • United States, Multiple Locations, Multiple Locations • Full-time

Posted 1mo ago

Job Location

United States, Multiple Locations, Multiple Locations

Tech Stack

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Infrastructure Engineer

About the job

The FIT Infrastructure team builds foundational accelerated compute platforms powering large-scale AI training and inference for Microsoft. Our mission is to deliver secure, reliable, and highly efficient GPU infrastructure for multi-tenant AI systems at a global scale, maximizing utilization, availability, performance, and reliability. This role is at the intersection of cloud infrastructure, distributed systems software, and frontier AI model serving, collaborating with various Microsoft teams to deliver end-to-end platform capabilities. As a Principal Software Engineer, you will work on mission-critical infrastructure directly powering large-scale AI systems, influencing the future of cloud GPU platforms and shaping long-term platform strategy.

Responsibilities

  • Develop systems for AI workload management, scheduling, preemption, and failover.
  • Develop systems for AI workload serving.
  • Design and build GPU accelerated infrastructure for training and inference workloads across bare metal, virtual machines, and containerized environments.
  • Build and operate advanced orchestration and resource governance using platforms like AKS and Kubernetes to enable fair sharing, isolation, and efficient utilization of accelerated resources.
  • Design, develop, test, and maintain services written in C# and Golang hosted on Kubernetes clusters.
  • Optimize performance, reliability, and utilization across large GPU fleets.
  • Drive end-to-end platform features from design through production, including observability and diagnostics.
  • Influence platform architecture and technical direction through design reviews and technical leadership.

Requirements

  • Bachelor's Degree in Computer Science or related technical field and 6+ years of technical engineering experience with coding in languages including C, C++, C#, Java, JavaScript, Linux, Go, or Python, or equivalent experience.
  • Proven ability to design and operate large-scale, production infrastructure with high reliability and performance requirements.
  • Demonstrated problem-solving skills and ability to debug complex, cross-layer systems issues.
  • Demonstrated technical leadership, including mentoring engineers and driving cross-team architectural alignment.
  • Hands-on experience with resource scheduling, preemption, and priority handling on large clusters.
  • Advanced collaboration and communication skills, with the ability to work across organizational boundaries.
  • Experience with large-scale online inferencing services.
  • Experience in building or operating multitenant AI platforms in cloud environments.
  • Familiarity with inference and/or distributed training stacks (e.g., NCCL collectives, model/data parallelism, LLMs).
  • Familiarity with GPU accelerated computing (e.g., CUDA, GPU drivers, device plugins, runtime integration).
  • DevOps experience with microservices architecture in a complex infrastructure and operational environment.

Benefits

  • Base pay range: $142,800 - $274,800 annually (US)
  • Base pay range: $188,000 - $304,200 annually (San Francisco Bay Area and New York City metropolitan area)
  • Certain roles may be eligible for benefits and other compensation.