Senior Software Engineer, Data - AI Portal
London, United Kingdom • Full-time
Posted 26d ago
About the job
Wayve is building the leading AI platform for autonomous driving, pioneering an end-to-end AI approach that enables vehicles to learn directly from real-world experience. As a Senior Software Engineer, Data, you will join a high-autonomy team focused on building Wayve’s Data Transfer Hub. This mission-critical infrastructure is designed to transfer petabytes of sensor data daily from global partners, forming the data foundations essential for large-scale AI model training. You will work with partners such as Uber, Stellantis, and Nissan, contributing to the core systems that power autonomous vehicles.
Responsibilities
- Build and operate large-scale pipelines for transferring video, LiDAR, and other sensor data from global partners.
- Shape the technical design of the Data Transfer Hub, balancing user experience, reliability, throughput, cost, and operational support.
- Write pipeline logic primarily in Python and manage infrastructure as code using Terraform.
- Collaborate with product, commercial, and partner-facing teams to understand constraints and solve problems.
- Take ownership of production systems, ensuring high engineering standards as requirements evolve.
- Participate in an out-of-hours on-call and support rotation once the service is established.
- Implement and evolve a globally distributed Data Transfer Hub using a hub-and-spoke model for data sharing at petabyte-per-day scale.
- Develop parallelized, distributed transfer pipelines using workflow orchestration tools like Flyte and event-driven patterns with Kafka.
- Build reliable and observable data systems, including retry strategies, backfills, data integrity checks, and alerting.
- Develop networking and large-scale file transfer solutions for cloud-to-cloud and cross-cloud connectivity, primarily using Azure.
- Improve the performance, efficiency, throughput, reliability, and cost-effectiveness of large-scale data transfer systems.
Requirements
- Experience building and operating large-scale data transfer, ingestion, or processing pipelines, ideally at multi-terabyte or petabyte scale.
- Strong software engineering skills and experience delivering and maintaining production systems.
- Proficiency in Python and Terraform.
- Experience with workflow orchestration tools (e.g., Flyte, Airflow, Prefect).
- Knowledge of building reliable, observable data systems, including retries, backfills, integrity checks, and alerting.
- Strong cloud platform skills, ideally with Azure, including experience with object/blob storage, large-scale file movement, and networking.
- A product mindset focused on customer needs and problem-solving.
- Comfort with evolving requirements, taking ownership, and clear communication.
- Experience working with external data partners or building customer-facing technical integrations (bonus).
- Background in autonomous vehicles, robotics, geospatial data, video processing, or other data-intensive domains (bonus).
- Experience with Kafka or other event-driven systems (bonus).
- Multi-cloud experience across Azure, AWS, and/or GCP (bonus).
- Experience improving cost efficiency, throughput, or reliability of large-scale cloud data systems (bonus).