Data Engineer

Big Data Engineer – Python, Spark, SQL

We are looking for a Big Data Engineer to develop end-to-end data pipelines in a Data Lake environment. This role involves designing and implementing data ingestion processes, writing scalable PySpark scripts, and working with SQL for data analysis and troubleshooting.

Key Responsibilities:

• Develop and maintain data pipelines in a Big Data environment.
• Build and optimize ETL/ELT processes for relational data systems.
• Write and debug Python scripts, with a focus on PySpark.
• Perform data analysis and troubleshooting using SQL.
• Implement and maintain CI/CD processes.
• Conduct unit and integration testing to ensure data integrity.
• Work with scheduling tools like CTRL-M.

Requirements:

• 2+ years of experience in Python development.
• 2+ years of experience working with Big Data technologies, focusing on Spark and Hive.
• Strong proficiency in writing and optimizing complex SQL queries.
• 3+ years of experience with Microsoft data technologies (SSIS, SQL Server).
• Familiarity with DWH, Data Lake, ETL, and ELT processes.
• Experience working with CI/CD pipelines.
• Advantageous skills:Experience with Teradata.
• Familiarity with Qlik Sense, Quilliup.
• Background in the banking/financial sector.

What We're Looking For:

• A proactive, independent thinker with strong organizational skills.
• A team player with excellent interpersonal skills.
• Someone eager to learn and grow, both technically and personally.

If you're passionate about Big Data and excited to work with cutting-edge technologies, we’d love to hear from you! 🚀