Senior Software Engineer - AI Compute, Together Cloud
$220k - $270k • Remote • San Francisco
Posted 1mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
Together AI is building the AI Native Cloud, an end-to-end platform for the full generative AI lifecycle, combining a fast LLM inference engine with state-of-the-art GPU cloud infrastructure. The Together Cloud team builds the flagship IaaS product that provides high-performance, AI-ready GPU clusters through a self-serve cloud console, along with the virtualized infrastructure layer powering Together's inference, RL, and fine-tuning products. As a Senior Software Engineer focusing on AI Compute, you will build and own major components of the next generation AI cloud platform – a highly available, global cloud infrastructure with cutting-edge virtualization of the latest ML hardware. This virtualized computing platform powers our own SaaS products and serves external cloud customers through self-serve offerings. The hard problems in rapidly scaling heterogeneous GPU fleets are software problems that we solve by building global and in-DC services, Kubernetes operators, and high-performance SDN libraries, forking hypervisors and Linux kernels, and building infra tailored to inference and fine-tuning. You'll own massive greenfield projects across their full lifecycle – scoping the problem, designing the system, and building it through to GA launch.
Responsibilities
- Build the GPU and network virtualization stack, including hypervisor, kernel, and SDN work for high-performance, portable, and isolated GPU compute and DC networking.
- Build and maintain the in-DC IaaS layer, including services, Kubernetes operators, and libraries for provisioning and managing compute, storage, and networks.
- Scale the distributed GPU scheduling and global management plane for on-demand and reserved clusters.
- Harden the monitoring and automated remediation layer for fault tolerance of distributed pretraining and large-scale inference.
- Own components end-to-end, including design, incremental shipping, and improving reliability.
- Raise the bar through code reviews, design feedback, and mentoring junior engineers.
- Build tooling for other teams, including testing frameworks, developer tools, and documentation, and contribute to the open-source Together AI platform.
Requirements
- 5+ years of professional software development experience, with strong proficiency in at least one backend programming language (Golang desired).
- Demonstrated ownership of large-scale projects driven end-to-end to completion, writing high-performance, well-tested, production-quality code.
- Experience building and operating high-performance and/or globally distributed micro-service architectures across one or more cloud providers (AWS, Azure, GCP).
- Strong systems knowledge across compute, networking, and storage, including concurrency, memory management, performant I/O, and scale.
- Excellent communication skills, able to write clear design docs and work effectively with technical and non-technical team members.
- Experience building and operating reliable, customer-facing production systems at scale, and owning infrastructure automation (Terraform, Ansible), observability (Prometheus, Grafana), and CI/CD (GitHub Actions, ArgoCD).