Senior Software Engineer, Data Infrastructure
$200k - $400k • San Francisco • FullTime
Posted 20d ago
About the job
Decagon is seeking a Senior Data Infrastructure Engineer to design, build, and operate the data systems that power the company's AI products. This role involves owning critical data pipelines and storage layers end-to-end, enhancing reliability and performance, and establishing streamlined processes for engineers to confidently work with data at scale. The ideal candidate will contribute to building and maintaining the foundational infrastructure for a leading conversational AI platform.
Responsibilities
- Design and implement high-throughput data pipelines and streaming systems with strong SLOs, clear runbooks, and actionable telemetry.
- Build and operate real-time and batch ingestion infrastructure using tools like Kafka, Flink, and Airflow.
- Own the analytical data layer, including schema design, query performance, and cost optimization across ClickHouse, BigQuery, or similar.
- Partner with research and product teams to architect data solutions, evaluate performance, and scale new features.
- Tune pipeline and query latencies by optimizing data paths, applying smart caching/partitioning, and hitting tight p95/p99 targets.
- Lead infrastructure-as-code (Terraform) and GitOps practices for data systems, reducing drift with reusable modules and policy-as-code.
- Participate in on-call rotations and drive down toil through automation and elimination of recurring data issues.
Requirements
- 5+ years building and operating production data infrastructure at scale.
- Hands-on experience with Tier 1 data technologies: ClickHouse, Kafka (or MSK/Pub-Sub/RabbitMQ), and Flink or dbt.
- Proven track record meeting high availability and low latency targets across streaming and batch workloads.
- Excellent observability skills (OpenTelemetry, Prometheus/Grafana, Datadog) and strong incident response discipline.
- Clear written communication and the ability to turn ambiguous data requirements into simple, reliable designs.
- Experience with CDC tooling (Debezium) and orchestration frameworks (Airflow, Dagster, or Prefect).
- Familiarity with Spark or Dask for large-scale data processing.
- Experience with cloud data warehouses (Snowflake, BigQuery, Redshift, Databricks).
- Experience being an early data/platform/infrastructure engineer at another company.
- Strong Kubernetes experience (GKE/EKS/AKS) and multi-cloud exposure (GCP, AWS, Azure).
- Experience with customer-managed deployments.
Benefits
- Equity
- Medical, Dental, and Vision benefits for you and your family
- Life Insurance and Disability Benefits
- Retirement Plan (e.g., 401K, pension)
- Parental Leave
- Fertility and family building benefits through Carrot
- Monthly stipend to support your wellness, lifestyle, and work-life balance
- Daily lunches and snacks in the office