Sr. Software Engineer - Ingestion Core team
$166k - $225k • Remote • San Francisco, California
Posted 26d ago
About the job
Databricks is seeking a Sr. Software Engineer to join the Ingestion Core Team. This team is responsible for making data ingestion simple, reliable, and efficient for all data types, including structured and unstructured. The role involves building distributed platform systems to incrementally ingest high-volume, petabyte-scale data from diverse sources into Delta Lake, ensuring schema evolution and change data capture. You will contribute to simplifying complex data challenges and empowering customers with analytics and machine learning tools.
Responsibilities
- Build distributed infrastructure for data ingestion, supporting streaming, incremental processing, and replication.
- Reduce end-to-end latency, increase throughput, and lower costs for data ingestion pipelines.
- Design and optimize streaming and distributed workloads for performance, cost, reliability, and scale.
- Explore and apply ML techniques to optimize streaming workloads.
- Develop monitoring and observability capabilities for ingestion workflows and systems.
- Collaborate with partner teams to enable use cases like RAG and AI agents.
- Build Agentic SDKs with strong evaluations.
Requirements
- 5+ years of experience writing production code in Java, Scala, Go, C++, or Python.
- Experience architecting, developing, and deploying large-scale distributed and asynchronous systems.
- Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.
- Experience building or operating systems where scale, throughput, latency, reliability, and cost are critical.
- Comfortable using AI tools and defining effective evaluations to shorten development loops.
Benefits
- Eligibility for annual performance bonus
- Equity
- Comprehensive benefits and perks