New: upload your resume and get matched to AI jobs. Free to start →

Engineering Director, Agentic AI Data Platform

$300k - $400k • US - San Francisco • Full-time

Posted 15d ago

Remote Work Policy

On-site

Employment Type

Full-time

Categories

AI Infrastructure Engineer

About the job

Airwallex is seeking a Director of Data Platform Engineering to lead the team responsible for the data backbone of their financial systems. This role involves setting the technical direction for high-volume financial data ingestion, managing their Databricks lakehouse, data orchestration, canonical financial models, and the underlying infrastructure to ensure reliability, scalability, and real-time processing. The position requires leading and growing a high-performing engineering team, providing mentorship, and fostering clear ownership. The director will also guide infrastructure strategy, drive performance and cost optimization, and partner with various teams to translate business needs into scalable data platform capabilities. A key aspect of the role is championing AI-assisted engineering practices to enhance team velocity while upholding high standards for code quality, testing, security, and reliability.

Responsibilities

  • Lead and grow a high-performing data platform engineering team, providing technical direction, mentorship, career development, and clear ownership.
  • Set the technical strategy for high-volume financial data ingestion across various financial systems.
  • Lead the evolution of the Databricks lakehouse using Spark, Delta Lake, Unity Catalog, dbt, Python, and SQL, optimizing for scale, performance, reliability, and cost.
  • Define standards for data contracts, schema evolution, drift detection, data quality, and canonical financial models.
  • Evolve the platform from batch toward near-real-time processing using Kafka, CDC, micro-batching, incremental processing, and Dagster orchestration.
  • Own the architecture of canonical financial models that transform complex source data into consistent financial objects and relationships.
  • Partner with infrastructure and platform teams to build reliable production foundations across observability, alerting, incident response, recovery, and operational readiness.
  • Guide infrastructure strategy across GCP, Terraform, Kubernetes/GKE, Helm, GitOps, and CI/CD.
  • Lead migrations from legacy pipelines to the lakehouse, with rigorous parity validation, staged rollouts, and clear rollback strategies.
  • Drive performance and cost optimisation across PostgreSQL, Databricks, and large-scale data workloads.
  • Partner with product, accounting, ML, and platform leadership to translate business needs into scalable data platform capabilities.
  • Set technical direction through architecture reviews, design documents, engineering standards, and long-term platform roadmaps.
  • Champion AI-assisted engineering practices that improve team velocity while maintaining high standards for code quality, testing, security, and reliability.

Requirements

  • 8+ years of experience in data platform, backend, or infrastructure engineering, including significant experience leading engineering teams.
  • Proven experience managing and developing senior engineers and technical leads and building strong, autonomous teams.
  • Deep expertise in data platforms, distributed systems, data modelling, and production data pipelines.
  • Strong experience with Python and SQL.
  • Hands-on experience with Spark, Databricks, Delta Lake, dbt, and modern lakehouse architectures.
  • Experience building reliable ingestion systems with large-scale backfills, incremental processing, checkpointing, rate-limit handling, idempotency, and failure recovery.
  • Strong understanding of streaming and event-driven architectures, including Kafka, CDC, and micro-batching.
  • Experience with data orchestration platforms such as Dagster and designing for freshness, partitioning, and incremental computation.
  • Experience optimising PostgreSQL and analytical workloads for performance and cost.
  • Experience operating production infrastructure on GCP, with familiarity with Terraform, Kubernetes/GKE, Helm, GitOps, and CI/CD.
  • Strong understanding of observability, reliability engineering, and production operations.
  • Excellent technical communication skills, with the ability to influence architecture and engineering decisions across teams.
  • Experience leading complex technical migrations and balancing long-term platform strategy with near-term business priorities.
  • Passion for building teams and infrastructure that power reliable, scalable, mission-critical financial systems.
  • Experience building data platforms for financial systems, accounting, payments, or other highly regulated products (preferred).
  • Experience applying AI-assisted engineering practices in production environments (preferred).