Principal Engineer - Perf and Benchmarking
$206k - $333k • Remote • Sunnyvale, CA / Bellevue, WA
Posted 2d ago
Job Location
Sunnyvale, CA / Bellevue, WA
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
CoreWeave is seeking a Principal Engineer to lead its Benchmarking & Performance team. This role will be responsible for a global performance data warehouse, ingesting, storing, transforming, and analyzing performance events across all data centers. You will also play a key part in industry-leading performance benchmarking publications, working closely with NVIDIA and the open-source community to demonstrate CoreWeave's performance and reliability leadership in AI.
Responsibilities
- Define the multi-year benchmarking strategy and roadmap, prioritizing models, workloads, and hardware tiers.
- Build, lead, and mentor a high-performing team of performance engineers and data analysts.
- Establish governance for claims, including documented methodologies, versioning, reproducibility, and audit trails.
- Lead end-to-end MLPerf Inference and Training submissions, including workload selection, cluster planning, runbooks, audits, and result publication.
- Coordinate optimization tracks with NVIDIA to achieve competitive results and drive upstream fixes.
- Design and implement a Kubernetes-native, repeatable benchmarking service to measure and report performance metrics across various configurations.
- Maintain a corpus of representative scenarios and datasets, automating comparisons across software releases and hardware generations.
- Build CI/CD pipelines and K8s controllers/operators to schedule benchmarks at scale.
- Integrate with observability stacks and results warehouses, and implement supply-chain integrity for benchmark artifacts.
- Partner with NVIDIA, key ISVs, and OSS projects to co-develop optimizations and upstream improvements.
- Support Sales/SEs with authoritative numbers for RFPs and competitive evaluations.
- Brief analysts and press with rigorous, defensible data.
Requirements
- 10+ years of experience building distributed systems or HPC/cloud services, with deep expertise in large-scale ML training or similar high-performance workloads.
- Proven track record of architecting or building planet-scale data systems.
- Deep understanding of GPU performance, model-server stacks, and distributed training frameworks.
- Proficiency with Kubernetes and ML control planes, including experience with SUNK, Kueue, and Kubeflow in production.
- Excellent communication skills to interface with executives, customers, auditors, and OSS communities.
Benefits
- Discretionary bonus
- Equity awards
- Comprehensive benefits program