Big Data Engineer

A leading, Global Consulting Company is seeking a Lead Data Engineer to join the team and lead the implementation of its vision to build Data Lake House & MLOps on the AWS cloud.

The Lead Big-Data Engineer will be part of the Generative AI revolution and build Generative AI products and services.

Responsibilities:

• Plan and build Data Lake House in the AWS cloud based on future technologies most companies will use in only 3-5 years.
• Have end to end ownership: Design, develop, deploy, measure and maintain the data platform.
• Solve production issues in the framework code.
• Collaborate with 40+ data engineers/scientists, understand their functional needs and build products to accelerate their productivity and quality of work.
• Influence directly on the way millions of people consume insurance and financial services.
• Research and build Generative AI services and applications.
• Lead and mentor data engineers, providing guidance and fostering a culture of continuous learning and professional development.
• Stay up to date with emerging technologies, industry trends, and best practices related to new technology, and proactively recommend their adoption when appropriate.
• Participate in the software engineering, processes and standards with an impact on the whole delivery organization.
• Communicate complex architectural concepts and recommendations effectively to both technical and non-technical stakeholders.

Tech Stack:

Python, Java, Spark, Kafka, Parquet, IceBerg, Avro, AWS S3, AWS Glue, AWS Athena, AWS DataZone, AWS Lake Formation, AWS SageMaker, AWS DocumentDB, AWS ElasticCahe, AWS OpenSearch, AWS DRS, AWS QuickSight, AWS EKS, AWS Managed Airflow.

Requirements:

· Previous experience in data engineering and familiarity with AWS databases services like RDS, SPARK, Glue, OpenSearch and more.

· 4+ years of experience developing large scale distributed data systems and services, experience in Big Data tools.

· 4+ years of working experience with AWS Cloud and services.

· 4+ years of expertise in the Software design and architecture process.

· Experience with SQL and No-SQL.

· 3+ years of software or data engineering development experience in one or more of the following programming languages: Python, Java, or Scala.

· Production systems understanding.

· Has good understanding and experience of CI/CD practices and Git, including MLOps practices.

· Proven experience in gathering and converting internal customer requirements into working solutions for large enterprises.

· Engineering leadership capabilities such as leading engineering discussions, presenting to other engineers and cross-functional leader.

· A helpful, can-do attitude and a willingness to take ownership of problems.