Lead Data Engineer

We are looking for a Lead Data Engineer to join our CTO Developer Experiences Platform Team to lead the implementation of our vision to build Data Lake House & MLOps on the AWS cloud.

Our vision is to reduce our engineers' toil by designing, building, and maintaining scalable, reliable, and efficient products (infrastructure and tools) that enable data engineers and data scientists to deliver high-quality data products quickly and securely.

On top of that our platform engineers are helping our product innovation team to build Generative AI experiences across our products and services. You have an amazing opportunity to be part of the Generative AI revolution and build Generative AI products and services.

We are looking for builders who are not afraid to explore the unknown and take big risks with new tech.

What are some of the things you do on a day-to-day basis?

• Plan and build Data Lake House in the AWS cloud based on future technologies most companies will use in only 3-5 years.
• Have end to end ownership: Design, develop, deploy, measure and maintain our data platform.
• Solve production issues in the framework code.
• Collaborate with 40+ data engineers/scientists, understand their functional needs and wants and build products to accelerate their productivity and quality of work.
• Influence directly on the way millions of people consume insurance and financial services.
• Research and build Generative AI services and applications.
• Lead and mentor data engineers, providing guidance and fostering a culture of continuous learning and professional development.
• Stay up to date with emerging technologies, industry trends, and best practices related to new technology, and proactively recommend their adoption when appropriate.
• Review current and future solutions, ask hard question and raise the bar on solutions designed by different delivery organizations.
• Participate in the software engineering, processes and standards with an impact on the whole delivery organization.
• Communicate complex architectural concepts and recommendations effectively to both technical and non-technical stakeholders.
• Help the in analyzing and adopting new technologies including assessing and embedding startup technologies.

Our Tech Stack

Python, Java, Spark, Kafka, Parquet, IceBerg, Avro, AWS S3, AWS Glue, AWS Athena, AWS DataZone, AWS Lake Formation, AWS SageMaker, AWS Document DB, AWS ElasticCahe, AWS OpenSearch, AWS DRS, AWS QuickSight, AWS EKS, AWS Managed Airflow.

What will you bring?

Tech Skills:

• Previous experience in data engineering and familiarity with AWS databases services like RDS, SPARK, Glue, OpenSearch and more.
• 5+ years experience developing large scale distributed data systems and services, experience in Big Data tools.
• 5+ years of working experience with AWS Cloud and services.
• 5+ years of expertise in the Software design and architecture process.
• Experience with SQL and No-SQL.
• Has at least 3 + years of software or data engineering development experience in one or more of the following programming languages: Python, Java, or Scala.
• Production systems understanding.
• Has good understanding and experience of CI/CD practices and Git, including MLOps practices.

Soft Skills:

• Boundaryless strategic thinker capable of driving high-visibility outcomes across the entire company.
• Proven experience in gathering and converting internal customer requirements into working solutions for large enterprises.
• Engineering leadership capabilities such as leading engineering discussions, presenting to other engineers, technical go-to person, and cross-functional leader.
• A helpful, can-do attitude and a willingness to take ownership of problems.
• Excellent problem-solving skills with a history of superb delivery against assigned tasks.