Member of Technical Staff - Training Platform
$150k - $300k • Remote • San Francisco • FullTime
Posted 3mo ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Prime Intellect is building the open superintelligence stack, a platform that unifies compute, environments, evaluations, secure sandboxes, high-performance training, and deployment for post-training at frontier scale. This role focuses on building the hosted training platform, enabling users to launch fine-tuning runs on managed GPU clusters with ease. The position spans both the developer-facing platform and the underlying Kubernetes-based training infrastructure, requiring a blend of AI/GPU knowledge, Kubernetes expertise, and strong programming skills.
Responsibilities
- Design and operate Kubernetes-based training and inference orchestration across multi-cluster, multi-cloud GPU fleets.
- Build and maintain Helm charts for composing trainers, inference servers, and supporting services.
- Develop Python control-plane agents to monitor pod status and synchronize clusters.
- Implement scheduling and autoscaling for heterogeneous hardware using KEDA, LeaderWorkerSet, and gang scheduling.
- Manage a GitOps workflow for all changes via PRs, Helm values, and CI.
- Build node-local model caches, checkpoint pipelines, and shared storage for efficient cold starts.
- Operate the observability stack (Prometheus, Grafana, Loki, DCGM) and facilitate GPU cluster debugging.
- Build developer-facing surfaces for hosted training, including job submission, live run monitoring, logs, and metrics.
- Develop FastAPI backend services and REST APIs to connect the platform to running clusters.
- Create real-time monitoring and debugging tools for streaming logs, step-level metrics, and failure analysis.
- Ship product UI using Next.js / React / TypeScript with shadcn, Tailwind, tRPC, and TanStack Query.
- Interface with RL trainers, inference servers, and environment servers within clusters.
- Productize new training capabilities, including new model architectures, RL algorithms, and modes.
Requirements
- Strong working knowledge of the modern AI stack, including open model families, finetuning techniques (LoRA, QLoRA, full FT, RLHF/RLAIF), and inference engines (vLLM, SGLang, TensorRT-LLM).
- Familiarity with GPU hardware tradeoffs (H100/H200/B200, NVLink, interconnects, memory hierarchy) and their impact on workloads.
- Understanding of distributed training fundamentals (data/tensor/pipeline/expert parallelism, NCCL, multi-node scheduling).
- Awareness of frontier AI trends and the ability to translate them into product decisions.
- Strong Kubernetes operations experience, including Helm, CRDs, operators, KEDA, gang scheduling, and GPU operator.
- Comfortable debugging production Kubernetes clusters (kubectl, pod lifecycle, node issues, networking).
- Cloud platform experience, preferably GCP (GCS, GKE, Cloud Run, Cloud Tasks).
- Experience with infrastructure automation (Helm, Terraform, Ansible) and a GitOps mindset.
- Proficiency with observability tools: Prometheus, Grafana, Loki, OpenTelemetry, DCGM.
- Solid Linux fundamentals, including networking, namespaces, and performance tuning.
- Strong Python backend development skills (FastAPI, async, SQLAlchemy).
- Experience building Python control-plane agents that interact with Kubernetes APIs.
- Proficiency in modern frontend development (TypeScript, React/Next.js, Tailwind, shadcn) for end-to-end product shipping.
- Experience with REST and tRPC API design.
- Experience building developer tools, dashboards, and live-monitoring UIs.
Benefits
- Cash compensation $150K–$300K
- Significant equity
- Flexible work arrangement (remote or San Francisco office)
- Full visa sponsorship and relocation support
- Professional development budget for courses and conferences
- Regular team off-sites and conference attendance
- Opportunity to shape the future of decentralized AI development