Software Engineer - Observability
Remote • San Francisco • FullTime
Posted 1mo ago
About the job
Baseten is seeking talented and experienced Software Engineers to join our Observability team within the Infrastructure organization. As an early member of this team, you will be pivotal in building and shaping the observability experience for our internal and external customers, directly impacting the reliability and operational excellence of Baseten's product systems. As Baseten scales its infrastructure across different cloud providers and diverse hardware, the volume and complexity of operational data is growing significantly. This team is responsible for building high-throughput ingest pipelines, cost-efficient storage, and agentic diagnostic tools to ensure that issues can be detected, diagnosed, and resolved rapidly, even as the systems become more complex.
Responsibilities
- Design and build scalable telemetry ingest and storage pipelines for metrics, logs, and traces across Baseten’s multi-cloud infrastructure.
- Own and evolve core observability platforms, driving migrations and architectural improvements for reliability, cost reduction, and scalability.
- Build instrumentation libraries, SDKs, and integrations to enable engineering teams to emit high-quality telemetry.
- Drive alerting and SLO infrastructure for defining, monitoring, and responding to reliability targets with minimal noise.
- Partner with Inference, Product, and Infrastructure teams to ensure observability solutions meet their unique needs.
Requirements
- Deep experience in at least one observability signal area (metrics, logging, tracing, or error analytics) and familiarity with others.
- Understanding of high-throughput data pipelines, columnar storage engines, and telemetry data ingestion/querying tradeoffs at scale.
- Experience operating or building on observability platforms like Prometheus, Grafana, ClickHouse, or OpenTelemetry.
- Strong proficiency in at least one of Python, Rust, or Go.
- Excellent communication skills and enjoyment in partnering with internal teams to improve operational visibility and incident response.
- Interest in applying AI/LLMs to operational workflows such as automated root cause analysis, anomaly detection, or intelligent alerting.
- Excitement about building foundational infrastructure and comfort working independently on ambiguous, high-impact technical challenges.
Benefits
- Competitive compensation, including meaningful equity
- 100% coverage of medical, dental, and vision insurance for employee and dependents
- Flexible PTO policy including company wide Winter Break
- Paid parental leave
- Fertility and family-building stipend through Carrot
- Company-facilitated 401(k)
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.