Senior Data Engineer

A company that develops a platform to prevent fraud and identity theft is looking for a Senior Data Engineer.

Main responsibilities:

• Provide the direction of the data architecture. Determine the right tools for the right jobs.

• Manage end-to-end execution of high-performance, large-scale data-driven projects, including design, implementation, and ongoing maintenance.

• Optimize and monitor the team-related cloud costs.

• Design and construct monitoring tools to ensure the efficiency and reliability of data processes.

• Implement CI/CD for Data Workflows.

Requirements:

• 5+ Years of Experience in data engineering and big data at large scales.

• Extensive experience with data lakes, warehouses, and lakehouses (Snowflake, Delta Lake, Iceberg, BigQuery, Redshift).

• Hands-on experience with Kafka, RabbitMQ, or similar for real-time data processing.

• Strong programming skills in Python or at least one OOP language (e.g., Java, Scala).

• Experience in working with any of the major cloud providers: Azure, Google Cloud , AWS)

• Hands-on experience in designing, building, and maintaining batch and streaming data pipelines using Python/PySpark for large-scale workloads.

• Familiar with the concepts of containerization and orchestration, with proficiency in Docker and Kubernetes.

• SW development background .

• Expertise in ETL development, data modeling, and data warehousing best practices.

• Knowledge of monitoring & observability (Datadog, Prometheus, ELK, etc) .

• Experience in working with transactional data lake formats like Delta Lake and Iceberg, including schema evolution, partitioning strategies, and performance tuning.

• Proven experience in developing data-driven microservices for processing and managing data.

• Experience with infrastructure as code, deployment automation, and CI/CD.

• practices using tools such as Helm, ArgoCD, Terraform, GitHub Actions, and Jenkins.

• Experience in Design Patterns concepts.