Senior Data Engineer

תאור משרה :Who We Are OriginAI, is a large AI research center that was established to address challenges at the forefront of AI research at national scale. The research is focused in the areas of Vision, Speech and NLP with the aim of producing world-class groundbreaking capabilities in these areas We work in hybrid mode and our new offices located near Tel-Aviv’s Savidor train station :Job Description We are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will have expertise in designing and optimizing large-scale data pipelines using RabbitMQ or Kafka, managing vector databases such as Vespa.AI, Elasticsearch, or other leading solutions, and developing Python-based workloads that integrate machine learning models. The systems will handle billions of items and operate in large cluster environments :Responsibilities Design, implement, and optimize data pipelines using RabbitMQ or Kafka to handle billions of items Ensure scalability, fault tolerance, and high-throughput processing capabilities in large clusters Develop Python-based ML modules integrated with the data pipelines Set up, maintain, and optimize a vector database (e.g., Vespa.AI, Elasticsearch, or similar) in a distributed cluster environment. Manage database performance, backups, and resource allocation to support billions of data items Monitor and troubleshoot system performance to ensure operational excellence Working with Dockerized environments :Requirements Proven experience with RabbitMQ or Kafka for building robust, large-scale data pipelines Proficiency in Python, with experience in creating adapters and integrating ML models Hands-on experience managing and optimizing vector databases such as Vespa.AI, Elasticsearch, or comparable solutions Expertise in working with distributed systems and large clusters Strong problem-solving skills with a focus on system performance and reliability at scale