Senior Software Engineer - AI Infrastructure Performance Insights & Observability
$182k - $242k • Remote • Sunnyvale, CA / Bellevue, WA
Posted 5d ago
Job Location
Sunnyvale, CA / Bellevue, WA
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Infrastructure Engineer
About the job
CoreWeave is seeking a Senior Engineer to join its Benchmarking & Performance team. This role will focus on building performance insights and observability systems for AI infrastructure, covering everything from individual GPUs and interconnects to distributed training and inference workloads. You will be responsible for detecting, diagnosing, and surfacing performance signals across global data centers, transforming raw telemetry into actionable insights for engineers, product teams, and executives. This is a hands-on role focused on building observability tooling that provides near real-time answers about infrastructure performance, rather than a traditional data engineering or BI position.
Responsibilities
- Design and build systems to assess AI infrastructure health and performance, including GPU utilization, interconnect behavior, and hardware degradation.
- Develop detection and diagnosis logic to identify anomalies and regressions before they cause incidents.
- Own and extend the time-series database (TSDB) layer for real-time observability.
- Write and optimize PromQL/MetricsQL queries for alerting, anomaly detection, and trend analysis.
- Build and validate pipelines and metrics for network fabric and GPU-level behavior observability.
- Design and build a performance data lake using formats like Apache Iceberg, Parquet, and Avro.
- Profile and tune query engines against columnar and time-series stores to meet latency and freshness SLAs.
- Build self-service views in tools like Grafana or Looker as a secondary deliverable.
Requirements
- 5+ years of experience building distributed systems, observability platforms, or performance engineering tooling, preferably for infrastructure or ML systems.
- Strong coding skills in Python or Go (C++ is a plus).
- Deep familiarity with networked systems, GPU infrastructure, and performance analysis.
- Hands-on experience with production-scale Kubernetes, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry).
- Working knowledge of time-series databases and fluency in PromQL or MetricsQL.
- Familiarity with data lake architectures and modern table formats (Iceberg, Parquet, Avro).
- Comfort working closely with hardware and workload behavior, including GPU utilization and interconnect fabric health.
- Strong communication and collaboration skills.