Staff Engineer, Distributed Storage and HPC & AI Infrastructure
$250k - $330k • Remote • San Francisco
Posted 6d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
In this role, you will operate, scale, and optimize multi-petabyte storage systems purpose-built for the world’s largest AI training and inference workloads. You’ll manage and scale high-performance parallel filesystems and object stores, evaluate and integrate cutting-edge technologies, and solve complex engineering challenges of operating at extreme throughput, low-latency data paths, and massive cluster-scale storage operations. You will also build Kubernetes-native storage operators and self-service platforms that provide automated provisioning, strict multi-tenancy, performance isolation, and quota enforcement at cluster scale.
Responsibilities
- Architect and implement the storage strategy and roadmap for Together AI, driving high-performance architectural decisions.
- Engineer and scale multi-petabyte AI/ML storage systems, integrating technologies like Vast, Weka, and Ceph, and optimizing costs through automated tiering.
- Develop intelligent caching and tiered storage architectures for extreme IOPS and cluster-wide throughput at GPU scale.
- Tune storage isolation at network layers to ensure secure, production-grade multi-tenancy.
- Code Kubernetes storage operators and controllers for automated provisioning, self-service, and quota enforcement.
- Engineer end-to-end data paths to achieve 10+ GB/s per GPU node, architect multi-tier caching, and tune parallel filesystems.
- Optimize end-to-end data paths through advanced benchmarking and profiling, contributing to open-source storage projects and internal tooling.
Requirements
- 8+ years in storage engineering, managing distributed storage at multi-petabyte scale.
- Proven track record deploying and operating high-performance storage for GPU/HPC clusters.
- Deep Kubernetes and cloud-native storage experience in production environments.
- Strong coding skills in Go and Python for building production-grade systems and tooling.
- BS/MS in Computer Science, Engineering, or equivalent practical experience.
- History of technical leadership in designing systems that improved performance, reliability, or cost efficiency.
- Deep expertise in distributed storage systems (Ceph, WekaFS, Lustre, Vast, GPFS, or similar).
- Production experience with object storage (S3, MinIO, Ceph, R2) including performance optimization.
- Experience with Kubernetes storage (CSI drivers, StatefulSets, PersistentVolumes, operators).
- Experience with storage optimization for GPU workloads, RDMA/InfiniBand networking, and parallel filesystem optimization.
- Experience with Infrastructure as Code tools like Terraform, Ansible, Helm, GitOps.
- Advanced knowledge of the Linux storage stack (filesystems, LVM, NVMe optimization).
- Experience with observability tools (Prometheus, Grafana, Thanos).
Benefits
- Competitive compensation
- Startup equity
- Health insurance
- Flexibility in terms of remote work