Staff Software Engineer - AI Compute, Together Cloud
$260k - $300k • Remote • San Francisco
Posted 24d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
Together AI is building the AI Native Cloud, an end-to-end platform for the full generative AI lifecycle, combining the fastest LLM inference engine with state-of-the-art GPU cloud infrastructure. The Together Cloud team builds the flagship IaaS product that provides high-performance, AI-ready GPU clusters through a self-serve cloud console, along with the virtualized infrastructure layer powering Together's inference, RL, and fine-tuning products. As a Staff Software Engineer focusing on AI Compute, you will set technical direction for and build major components of the next generation AI cloud platform – a highly available, global cloud infrastructure with cutting-edge virtualization of the latest ML hardware. This is an architect-and-build role where you will set the architecture for automated bootstrapping of GPU data centers, high-performance virtualization, and fault-tolerant decentralized control planes, and be a key owner of the hardest parts, in the code as well as the design. Your designs will span the IaaS layer up to the global management plane that schedules capacity across all data centers. At this level, the job is as much leverage as code: the standards you set and the engineers you grow decide how fast the rest of Together Cloud ships.
Responsibilities
- Own the GPU and network virtualization stack, including hypervisor, kernel, and SDN work for high-performance, portable, and isolated GPU compute and DC networking.
- Own the in-DC IaaS layer, architecting and roadmapping services, Kubernetes operators, and libraries for provisioning and managing compute, storage, and networks.
- Design the GPU scheduling and global management plane, including the distributed control plane for on-demand and reserved clusters.
- Architect monitoring and automated remediation strategies for fault tolerance, focusing on automated detection, isolation, and recovery of failed nodes.
- Set technical direction across teams by leading design reviews, resolving architectural disagreements, and unblocking dependencies.
- Grow the team by mentoring engineers and deepening expertise in virtualization, DC networking, and GPU infrastructure.
- Set the engineering bar by creating testing frameworks, tools, and documentation to ensure system robustness and usability.
Requirements
- 7+ years of professional software development experience with expert-level proficiency in at least one backend language (Golang desired), writing high-performance, well-tested, production-quality code.
- Track record of owning the architecture of large distributed systems from inception to production at scale.
- Deep experience building and operating globally distributed, high-performance microservice architectures across cloud providers (AWS, Azure, GCP).
- Expert systems knowledge across compute, networking, and storage, including concurrency, memory management, performant I/O, and global scale.
- Demonstrated technical leadership beyond individual contributions, including mentoring senior engineers and driving cross-team alignment.
- Excellent communication and diplomacy skills, with the ability to write effective design documents and collaborate with diverse stakeholders.
- Experience building and operating reliable, customer-facing production systems at scale, including infrastructure automation, observability, and CI/CD.