Member of Technical Staff - Inference
$150k - $300k • Remote • San Francisco • FullTime
Posted 3mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
LLM Engineer
About the job
Prime Intellect is building the open superintelligence stack, a platform designed to provide AI labs with the infrastructure they need to develop advanced AI models. Their platform, Lab, unifies compute, environments, evaluations, secure sandboxes, and high-performance training and deployment for post-training at scale, including SFT, RL, tool use, agent workflows, and continuous model improvement. The company is focused on creating open-source models trained for long-horizon tasks and the full-stack platform required for their development. They are seeking individuals passionate about building at the intersection of frontier research, real infrastructure, and go-to-market strategies for a nascent category.
Responsibilities
- Build a multi-tenant LLM serving platform across cloud GPU fleets.
- Design placement and scheduling algorithms for heterogeneous accelerators.
- Implement multi-region/zone failover and traffic shifting for resilience and cost control.
- Develop autoscaling, routing, and load balancing to meet throughput/latency SLOs.
- Optimize model distribution and cold-start times across clusters.
- Integrate and contribute to LLM inference frameworks like vLLM, SGLang, TensorRT-LLM.
- Optimize configurations for parallelism, memory management, and other performance axes.
- Profile kernels, memory bandwidth, and transport; apply techniques like quantization and speculative decoding.
- Develop reproducible performance suites for latency, throughput, context length, batch size, and precision.
- Embed and optimize distributed inference within the RL stack.
- Establish CI/CD pipelines with artifact promotion and performance gates.
- Build metrics, logs, and tracing for structured incident response and SLO management.
- Document architectures, playbooks, and API contracts; mentor and collaborate cross-functionally.
Requirements
- 3+ years building and running large-scale ML/LLM services with clear latency/availability SLOs.
- Hands-on experience with at least one of vLLM, SGLang, TensorRT-LLM.
- Familiarity with distributed and disaggregated serving infrastructure (e.g., NVIDIA Dynamo).
- Deep understanding of inference internals (prefill vs. decode, KV-cache, batching, sampling, parallelism strategies).
- Comfortable debugging CUDA/NCCL, drivers/kernels, containers, service mesh/networking, and storage.
- Proficiency in Python for systems tooling and backend services.
- Experience with PyTorch for LLM inference engine development and integration.
- Experience with cloud platforms (AWS/GCP) and cloud deployment patterns.
- Experience running infrastructure at scale with containers on Kubernetes.
- Knowledge of GPU architecture, CUDA runtime, NCCL, InfiniBand, and GPU-aware scheduling.
- Familiarity with CUDA/Triton kernel development and Nsight Systems/Compute profiling (nice to have).
- Experience with systems performance languages like Rust or C++ (nice to have).
- Familiarity with data and observability tools (Kafka, Redis, gRPC, Prometheus, Grafana, OpenTelemetry) (nice to have).
- Experience with infrastructure automation tools (Terraform, Ansible) (nice to have).
- Contributions to open-source serving, inference, or RL infrastructure projects (nice to have).
Benefits
- Cash compensation range of $150-300k
- Significant equity incentives
- Flexible work arrangement (remote or San Francisco office)
- Full visa sponsorship and relocation support
- Professional development budget
- Regular team off-sites and conference attendance
- Opportunity to shape decentralized AI and RL