Senior Data Engineer
Salary not listed • Hyderabad, Telangana, India • Full-time
Posted 1mo ago
About the job
We are seeking a Senior Data Engineer to join our team and contribute to the design and delivery of our cloud-native healthcare data platform. This role involves implementing scalable data solutions on AWS, utilizing technologies like Apache Iceberg, Lake Formation, Glue Catalog, Athena, and dbt, along with modern orchestration frameworks. You will collaborate closely with platform, analytics, and business teams, combining strong hands-on engineering with cross-functional teamwork to drive technical execution from design through production deployment.
Responsibilities
- Deliver complex data engineering projects in collaboration with cross-functional teams.
- Drive technical execution from design through production deployment.
- Implement scalable data patterns and reusable frameworks.
- Design and implement batch and near-real-time data pipelines.
- Build reusable ingestion, transformation, validation, and publishing frameworks.
- Support the modernization of legacy data workloads.
- Contribute to Apache Iceberg implementation and optimization, including schema evolution, partitioning, compaction, and metadata management.
- Ensure efficient data storage and query performance.
- Implement data quality frameworks and validation layers.
- Support observability and monitoring practices for data systems.
- Contribute to operational excellence and reliability improvements.
- Participate in architecture and design discussions.
- Conduct and participate in code reviews.
- Mentor junior engineers and share best practices.
- Oversee specific business processes within the Information Security Management System (ISMS).
- Manage ISMS documentation, conduct risk assessments, and implement risk treatment plans.
- Conduct Business Continuity Planning (BCP) and other information security continuity tests.
- Monitor and report on the performance of the ISMS.
- Implement security policies and procedures and report any issues.
Requirements
- 5+ years of Data Engineering experience.
- Experience contributing to enterprise-scale data initiatives.
- Proficiency with AWS services including S3, Glue, Athena, and Lake Formation.
- Experience with IAM and security controls.
- Experience with Apache Iceberg (preferred), Delta Lake, or Hudi.
- Experience with dbt.
- Experience with Airflow or AWS Step Functions.
- Proficiency in Python.
- Proficiency in SQL.
- Experience with PySpark or Spark.