Data Platform Engineer
Remote • Remote - Americas
Posted 4d ago
About the job
Alpaca is a global leader in agent-first brokerage infrastructure, serving hundreds of financial institutions worldwide with institutional-grade APIs. Our mission is to open financial services to everyone, supported by a diverse team of experienced professionals committed to open-source contributions and fostering a vibrant community. We are backed by significant funding from top-tier investors and are looking for passionate individuals to join our dynamic, globally distributed team.
The Data Platform team builds and operates the shared infrastructure for data storage, movement, and consumption across Alpaca, processing hundreds of millions of events daily. As a Data Platform Engineer, you will build and operate reliable systems across this platform, contributing to our distributed query, lakehouse, CDC, and real-time streaming infrastructure. You will own projects from implementation to production, enhancing the platform's scalability, reliability, and developer experience.
Responsibilities
- Design, develop, and evolve core data platform components including distributed query engines, orchestration, lakehouse storage, metadata, and cataloging.
- Own lakehouse infrastructure as code, building reliable deployment workflows with Terraform and Ansible on Kubernetes.
- Build and operate scalable, low-latency streaming and CDC pipelines, and dependable batch ingestion paths into Apache Iceberg.
- Develop and scale serving and BI infrastructure for performant, governed, self-service access to lakehouse data.
- Ensure platform reliability through observability, alerting, on-call practices, incident response, and service-level objectives.
- Partner with DevOps, Analytics Engineering, and stakeholders to address infrastructure gaps, define technical requirements, and deliver reusable platform capabilities.
Requirements
- 4+ years of experience in software engineering, data infrastructure, or platform engineering with ownership of complex production systems.
- Strong understanding of distributed systems and production experience with query engines like Trino or Presto.
- Strong experience operating Kubernetes-based data infrastructure using infrastructure-as-code and deployment tools (Terraform, Helm, Ansible, Argo CD).
- Experience designing and operating multi-cloud compute platforms, balancing workload locality, demand, resource isolation, and cost-performance.
- Hands-on experience designing and operating cloud lakehouse architectures (GCP, AWS, Azure) with object storage, metastores, and open table formats (Apache Iceberg).
- Experience operating large-scale streaming and CDC systems (Kafka, Redpanda, Debezium).
- Strong Python and SQL skills, with experience building data pipelines and platform tooling using orchestration and ingestion frameworks (Airflow, Airbyte).
- Strong operational judgment, ability to make sound technical decisions, communicate trade-offs, and work through ambiguity.
Benefits
- Competitive Salary & Stock Options