Data Platform Engineer

We’re seeking an experienced Software Engineer to join our Data Platform team. In this role, you will help design, build, and maintain a scalable data infrastructure capable of processing and analyzing massive volumes of events daily. This platform will underpin advanced cybersecurity research, supporting both batch and real-time data pipelines, complex data modeling, machine learning workflows, and large language models (LLMs).

Key Responsibilities:

• Data Architecture & Development: Design and implement scalable data systems including data lakes, warehouses, and pipelines.
• Data Integration: Build and manage pipelines for ingesting and processing large, diverse datasets while ensuring quality and consistency.
• Workflow Orchestration: Automate and oversee data workflows using modern orchestration tools.
• Database Management: Optimize database performance through indexing, partitioning, aggregations, and high-performance queries.
• Scalability & Performance: Ensure systems handle high-volume data efficiently while maintaining reliability.
• ML & LLM Support: Collaborate with data scientists to integrate and optimize machine learning and LLM workflows, including real-time RAG pipelines and ML-powered features.
• Cross-Team Collaboration: Work closely with engineers, data scientists, and cybersecurity specialists to align the platform with research and business objectives.
• Innovation & Trends: Stay up-to-date with emerging technologies in big data, cloud computing, and cybersecurity to continuously improve the platform.

Required Skills & Experience:

• Proven experience designing and building large-scale data systems with cloud technologies (AWS, GCP, Azure).
• Strong backend development skills for complex, data-intensive applications.
• Hands-on experience in advanced data modeling and schema design for batch and real-time processing.
• Experience with streaming data technologies (Kafka, Flink, Spark Streaming, or similar).
• Deep understanding of modern databases (SQL, NoSQL, NewSQL, graph databases) and query optimization.
• Knowledge of data security, encryption, and governance best practices.
• Strong coding standards with experience in CI/CD and test-driven development.
• Experience with microservices, Kubernetes, Terraform – a plus.
• Familiarity with data orchestration tools like Dagster or Apache Airflow – a plus.