Member of Technical Staff, Sandbox Infrastructure
$275k - $315k • San Francisco • FullTime
Posted 1mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
SF Tensor is building the future of high-performance compute for AI by rethinking and rebuilding the entire stack, from hardware to cloud infrastructure. They aim to eliminate the friction and cost associated with compute, making it faster, cheaper, and more accessible. Their core products include the Kernel Optimizer, which automatically finds the fastest code configurations for various hardware, and the Model Foundry, which streamlines research runs and manages workloads across different clouds and chips. This role focuses on building a critical layer for their compiler's search process: a serverless GPU container service that can run untrusted, freshly generated kernels quickly and safely across multiple vendors and accelerator types.
Responsibilities
- Extend the sandboxing stack to support new vendors and accelerators.
- Build and maintain GPU virtualization at the driver and ioctl level, including gVisor integration.
- Implement preemption-aware scheduling, checkpointing, and rescheduling for sandboxes on spot capacity.
- Support multi-GPU and multi-node sandboxes, including interconnects like NVLink and RDMA paths.
- Own the end-to-end live migration process, including socket-preserving migration.
- Guarantee measurement and profiling fidelity and isolation within sandboxes.
- Collaborate with compiler, post-training, and kernel teams to optimize throughput.
Requirements
- Strong low-level systems engineering background (Linux kernel, containers, namespaces, cgroups, syscall interception, hypervisors).
- Experience in GPU systems engineering (drivers, runtimes, accelerator fleet scheduling).
- Comfort with distributed systems failure modes (preemption, partial failure, checkpoint/restore).
- Proficiency in Go, C/C++, or Rust.
- A proactive approach to building custom solutions when vendor support is lacking.