Senior Data Engineer

MeteorOps is a top DevOps consulting firm. We help companies and startups implement cutting edge DevOps & cloud solutions.

We are seeking a highly skilled Senior Data Engineer with expertise in Apache Airflow. The ideal candidate will have extensive experience in developing, managing, and optimising complex workflows using Apache Airflow. This role requires a deep understanding of workflow orchestration, data pipeline management, and the ability to troubleshoot and optimize existing systems. The Apache Airflow Expert will work closely with our data engineering, data science, and DevOps teams to ensure seamless data flow and integration across our platforms.

• Design and Develop Workflows: Author, schedule, and monitor workflows using Apache Airflow.
• Optimise Pipelines: Enhance and optimize existing data pipelines for performance and reliability.
• Maintain and Support: Provide ongoing support and maintenance for Airflow environments, including troubleshooting issues and implementing fixes.
• Best Practices Implementation: Establish and enforce best practices for workflow management and data pipeline development.
• Integration: Collaborate with data engineering and data science teams to integrate Airflow with various data sources and sinks.
• Documentation: Create and maintain comprehensive documentation for all workflows and pipelines.
• Training and Mentorship: Train and mentor junior engineers on the best practices and use of Apache Airflow.
• Stay Updated: Keep up to date with the latest features and updates in Apache Airflow and related technologies.

• Education: Bachelor's or Master’s degree in Computer Science, Information Technology, or a related field.
• Experience:
• Proven experience with Apache Airflow in a production environment.
• Strong background in data engineering and workflow orchestration.
• Technical Skills:
• Proficiency in Python and SQL.
• Experience with cloud platforms (AWS, GCP, Azure) and their data services.
• Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes.
• Knowledge of CI/CD pipelines and tools such as Jenkins or GitLab CI.
• Experience with version control systems like Git.
• Soft Skills:
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills.
• Ability to work in a fast-paced, dynamic environment.

• Certifications: Apache Airflow certification, such as from Astronomer.
• Additional Experience:
• Experience with other workflow orchestration tools like Luigi or Prefect.
• Familiarity with data warehousing solutions like Snowflake, Redshift, or BigQuery.
• Knowledge of monitoring and alerting tools such as Prometheus, Grafana, or ELK stack.