Data Engineer

About the job

Responsibilities

• Build, operate, and improve data pipelines for analytics and model training.
• Deliver reliable internal analytics for data-driven decision-making across teams.
• Structure and document long-horizon agentic data into usable datasets for analysis and training.
• Select and justify ETL/ELT approaches based on destination capabilities, reproducibility, latency, and cost.
• Manage production pipeline operations, including failures, re-runs, backfills, and debugging across various components.
• Design and adapt schemas, storage formats, and data models to evolving needs.
• Develop memory-efficient, scalable processing using streaming and partitioning, identifying when single-machine workloads are insufficient.
• Communicate data location, lineage, quality, limitations, and fitness for use to both technical and non-technical stakeholders.

Requirements

• 4+ years of data engineering experience in building and operating production data pipelines.
• Strong SQL, data modeling, and practical schema design/evolution skills.
• Deep understanding of storage formats and database types, and their appropriate use cases.
• Familiarity with the data stack layers: ingestion, storage, transformation, orchestration, cataloging, and serving.
• Proficiency in Python and solid shell scripting for batch and storage operations.
• Hands-on cloud experience with object storage, batch compute, and IAM.
• Production experience with at least one workflow orchestration system (e.g., Airflow, Dagster, Prefect).
• A strong focus on data quality, ensuring data accuracy and reliability.
• Extensive use of AI-assisted development tools with disciplined review of generated code.