Data Engineer
Salary not listed • London • Full-time
Posted 1mo ago
About the job
We are seeking a Data Engineer to construct the ingestion and search systems that power our AI products. Our data sources are diverse, including global patent literature, case law, technical standards, scientific databases, academic papers, and web content, spanning structured records, documents, images, audio, and video. You will be responsible for both bulk ingestion and on-demand retrieval, ensuring this information is searchable and integrated into our products. You will own systems from initial source acquisition through to serving queries, working closely with AI researchers and product engineers with significant autonomy in system design and implementation.
Responsibilities
- Build and operate high-throughput, resumable pipelines for large datasets with efficient incremental updates, monitoring, and failure recovery.
- Extract useful content from complex documents and various formats, handling malformed records and schema changes while validating outputs and preserving structure/metadata.
- Develop keyword, vector, and structured search capabilities, designing schemas, indexes, and partitioning for fast queries over millions of records.
- Link information across diverse sources, preserving dates, versions, and ensuring traceability to original sources.
- Profile parsing, ingestion, database builds, and queries, diagnosing and resolving CPU, memory, and storage I/O bottlenecks to optimize throughput, latency, and cost.
Requirements
- Strong Python and SQL skills, with experience designing and operating production databases.
- Solid experience building and operating production data pipelines over large, messy datasets.
- Expertise in running search systems over large document collections.
- Proven ability to take end-to-end ownership from raw data to user-facing functionality.
- Good understanding of schema design, indexing, and query optimization.
- Track record of diagnosing and fixing performance bottlenecks in live systems through profiling and measurement.
Benefits
- Competitive Salary
- Significant Equity
- Full visa sponsorship
- Private medical insurance
- Free meals