Data Engineer

About AllCloud

AllCloud is a leader in amplifying organizations’ cloud potential through AI. With a track record of hundreds of successful migrations and implementations across AWS and Salesforce, AllCloud has developed strategies and solutions that enable businesses of all sizes to remain at the forefront of innovation.

AllCloud is a leader in AI-led professional and managed services. As an AWS Premier and audited managed services Partner, and Salesforce Consulting partner, AllCloud provides comprehensive AI-led cloud journey support, from initial migration to ongoing management through our Engage Managed Services. Our expertise ensures that clients remain aligned with ecosystem best practices while focusing on their core business growth.

AllCloud serves clients across the globe with offices in EMEA and North America. www.allcloud.io

Job Description

We are looking for a talented and experienced Data Engineer with a passion for data and cloud technologies to join our team. In this role, you will be a key player in designing, developing, and maintaining our Databricks-based data platform. You will bridge the gap between robust Data Engineering and high-performance Analytics Engineering.

The ideal candidate understands the Lakehouse architecture and has the ability to build scalable ETL pipelines that drive data-driven business decisions.

Key Responsibilities

• End-to-End Pipelines: Develop and maintain complex ETL/ELT pipelines using Python/PySpark on the Databricks platform.
• Data Architecture: Implement and manage data layers following the Medallion architecture (Bronze, Silver, Gold) using Delta Lake.
• Optimization & Analytics: Set up and manage Databricks SQL Warehouses, perform query optimization, and utilize internal visualizations for rapid data exploration.
• Data Modeling: Design data models (Star Schema / Snowflake) in the Gold layer to ensure optimal performance for BI tools (e.g., Power BI, Tableau).
• Data Governance: Manage metadata, permissions, and lineage using Unity Catalog.
• Quality Control: Implement automated Data Quality tests as an integral part of the CI/CD data pipelines.

Requirements

• Professional Experience: 2–4 years of experience as a Data Engineer – Must.
• Databricks Expertise: At least 1 year of intensive, hands-on experience with the Databricks platform (including Notebooks and Workflows).
• Development Skills:
• High proficiency in PySpark (or Spark Scala).
• Expertise in writing complex SQL (Window Functions, CTEs, Performance Tuning).
• Technical Knowledge: Practical experience with Delta Lake, file formats (Parquet), and working with Cloud Storage (Azure ADLS / AWS S3).
• Modeling: Proven experience in designing Fact and Dimension tables.

Advantages (Nice to Have)

• Experience with Delta Live Tables (DLT) or dbt (Data Build Tool) on Databricks.
• Familiarity with MLOps concepts and MLflow.
• Official Databricks Certification (Associate Developer / Data Engineer).
• Experience with external orchestration tools (e.g., Apache Airflow).

Soft Skills

• Ability to work independently and lead tasks end-to-end in an Agile environment.
• Strong troubleshooting skills for complex, distributed processes.
• Excellent communication skills and the ability to collaborate with various stakeholders (Analysts, Product Managers, and Data Scientists).

AllCloud is an Equal Opportunity Employer and considers applicants for employment without regard to race, color, religion, sex, orientation, national origin, age, disability, genetics or any other basis forbidden under federal, provincial, or local law.