Big Data Developer

Data Platform Engineer (Big Data & AI Infrastructure)

The Role

We are seeking a talented Software Engineer to join our Data Platform team. In this position, you will be a key architect in designing and implementing a high-scale data ecosystem capable of processing and analyzing massive volumes of daily events and connections.

This platform serves as the backbone for our advanced security research models, supporting a hybrid of real-time streaming and batch processing, complex data modeling, and the integration of Large Language Models (LLMs) and Machine Learning pipelines.

Core Responsibilities

• System Architecture: Design and develop a robust data environment, including the architecture of data lakes, warehouses, and sophisticated processing pipelines.
• Data Ingestion: Build and maintain scalable pipelines to ingest diverse, large-scale datasets while ensuring strict data integrity and consistency.
• Workflow Orchestration: Automate and manage complex data workflows using modern orchestration frameworks.
• Advanced Database Management: Lead the implementation and optimization of database services, focusing on high-performance indexing, partitioning, and query execution.
• Scale & Reliability: Ensure the platform remains performant and scales seamlessly under heavy, high-velocity loads.
• AI & LLM Integration: Partner with Data Scientists to integrate ML workflows and optimize RAG (Retrieval-Augmented Generation) pipelines for real-time production features.
• Cross-Team Collaboration: Align data architecture with business intelligence and research goals by working closely with engineering and security experts.

Requirements

• Large-Scale Data Expertise: Proven track record of building complex data architectures from the ground up using cloud providers (AWS, GCP, or Azure).
• Backend Engineering: Extensive experience in backend development for high-performance, data-centric systems.
• Advanced Data Modeling: Hands-on experience creating schemas and models for both real-time and batch processing environments.
• Streaming Technologies: Direct experience with high-velocity data streaming tools (e.g., Kafka, Flink, or Spark Streaming).
• Database Mastery: Deep understanding of SQL, NoSQL, and Graph databases, with a focus on performance tuning at scale.
• Security & Governance: Knowledge of data encryption and governance best practices to protect sensitive information.
• Software Excellence: Commitment to high code quality, robust testing strategies, and CI/CD principles.

Advantages

• Experience with Kubernetes (K8s) and microservices architecture.