Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)

About the job

Responsibilities

• Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
• Design and build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
• Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
• Build and operate platforms supporting analytics, machine learning, generative AI, and agentic AI workloads.
• Create reusable data products, services, APIs, and self-service capabilities.
• Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines.
• Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
• Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
• Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance.
• Apply security and privacy-by-design principles for sensitive and regulated data environments.
• Evaluate emerging data and AI technologies through prototypes and proof-of-concepts.
• Translate research or product requirements into practical architectures and incremental delivery plans.
• Provide technical leadership through architecture reviews, code reviews, troubleshooting, and mentorship.
• Partner with cross-functional teams to mature R&D initiatives into production solutions.
• Communicate technical decisions, trade-offs, risks, and roadmap recommendations.
• Drive adoption of modern data engineering practices, platform automation, and platform reliability.

Requirements

• 12-18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions.
• Extensive experience designing, building, and operating large-scale production data platforms.
• Expert programming skills in Python and PySpark, with modern software engineering practices.
• Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large-scale analytical processing.
• Strong hands-on experience with distributed data processing technologies like Apache Spark and modern lakehouse platforms.
• Experience building and operating cloud-native data platforms in AWS, Azure, or other enterprise cloud environments.
• Strong knowledge of modern data architecture patterns (data lakes, lakehouses, data meshes, data products, event-driven architectures).
• Experience with cloud storage, data integration, streaming, serverless, observability, and security services.
• Experience building scalable batch and real-time data pipelines.
• Experience with workflow orchestration platforms (Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar).
• Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
• Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
• Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges.
• Ability to make sound architectural decisions balancing delivery speed, innovation, maintainability, security, and cost.
• Proven ability to thrive in R&D and innovation-focused environments with evolving priorities.
• Strong communication, collaboration, technical leadership, and mentoring skills.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or equivalent practical experience.