Data Engineer

About The Position

About AllCloud

AllCloud is a leader in amplifying organizations’ cloud potential through AI. With a track record of hundreds of successful migrations and implementations across AWS and Salesforce, AllCloud has developed strategies and solutions that enable businesses of all sizes to remain at the forefront of innovation.

AllCloud is a leader in AI-led professional and managed services. As an AWS Premier and audited managed services Partner, and Salesforce Consulting partner, AllCloud provides comprehensive AI-led cloud journey support, from initial migration to ongoing management through our Engage Managed Services. Our expertise ensures that clients remain aligned with ecosystem best practices while focusing on their core business growth.

AllCloud serves clients across the globe with offices in EMEA and North America. www.allcloud.io

Job Summary

We are seeking a Data Engineer to join our team and contribute to cutting-edge AI projects. The role involves managing the ingestion of massive datasets, followed by in-depth data exploration and validation in close collaboration with Data Scientists. You will be responsible for the design and development of critical, large-scale, and diverse data pipelines across both Cloud and On-Prem environments.

Summary Of Key Responsibilities

• Data Pipeline Architecture: Designing and implementing robust, large-scale data pipelines across both Cloud (AWS) and On-Premises environments.
• AI & ML Collaboration: Partnering closely with Data Scientists to ingest, validate, and process massive datasets required for AI/ML model development.
• System Optimization: Enhancing data processing performance through advanced partitioning techniques and architectural optimizations.
• End-to-End Delivery: Managing the full data lifecycle—from initial ingestion and cleansing to production-grade deployment using CI/CD and orchestration tools.
• Cross-Functional Integration: Coordinating with development and production teams to ensure data solutions align with organizational needs and end-user requirements.
  
  

• At least 3 years of experience as a Data Engineer – Mandatory.
• 3 years of experience with Python – Mandatory.
• Hands-on experience with Spark for large-scale data processing – Mandatory.
• B.Sc./B.A. in Computer Science, Engineering, Mathematics, or Statistics – Mandatory.
• At least 2 years of practical experience with AWS services (Athena, Glue, Step Functions, EMR, Redshift, RDS) – Significant Advantage.
• Deep understanding of designing, developing, and optimizing complex solutions for high-volume data transfer and processing.
• Proficiency in optimization techniques and working with various data partitions/formats such as Parquet, Avro, HDF5, and Delta Lake.
• 3 years of experience with Object-Oriented Programming (OOP).
• Experience with Docker, Linux, CI/CD tools, and Kubernetes.
• Experience with data pipeline orchestration tools such as Airflow or Kubeflow.
• Solid understanding of Machine Learning concepts and workflows.
• Familiarity with GenAI / Prompt Engineering solutions – Advantage.
  
  

• Development teams, Production teams, and End-users.
  
  

• Self-motivated and independent with a strong ability to collaborate within a team.
• High self-learning capacity and strong analytical skills.
• Ability to perform under pressure with a high level of responsibility and commitment.