Senior Data Engineer (AI)

Senior Data Engineer – Advanced Technological Defense Organization

Role Overview

We are seeking a highly skilled and motivated Senior Data Engineer to lead the development of critical data infrastructures within a cutting-edge technological security organization.

This role involves building complex data systems that integrate diverse information sources (structured and unstructured) in secure, high-availability environments. In this position, you will have a direct impact on the organization’s ability to process, consolidate, enrich, and analyze data for vital operational and strategic needs.

Key Responsibilities

• Data Architecture & Design: Design and lead complex data architectures within high-security environments.
• Pipeline Development: Develop and maintain robust Data Pipelines for processing structured and unstructured data (text, video, images, logs, and signals).
• Database Management: Integrate multiple heterogeneous databases and lead the construction and management of Data Lake, Lakehouse, and Data Warehouse solutions.
• Real-Time Processing: Work with Near/Real-Time data streams to support immediate operational requirements.
• Reliability & Performance: Ensure data quality, traceability, integrity, and reliability while optimizing for high volumes and high availability.
• Technological Leadership: Set engineering standards and best practices, collaborating closely with AI, Research, Intelligence, and Product teams.

Mandatory Requirements

• Experience: +6 years of experience as a Data Engineer, including proven experience in complex/critical environments.
• Programming Proficiency: High proficiency in Python and/or Java/Scala.
• Deep experience in Data Modeling and SQL.
• Proven experience working with both structured and unstructured data.
• Storage & Databases: Hands-on experience with various storage systems:
• RDBMS: PostgreSQL, Oracle, or MySQL.
• NoSQL: MongoDB, Cassandra, or Elasticsearch.
• Object Storage: S3, GCS, or HDFS.
• ETL / Orchestration (Airflow, Prefect, Dagster) Experience with tools
• Streaming Data (Kafka / Pulsar / Kinesis) Processing experience
• Deep understanding of distributed architectures and Scalability-4. A significant advantage
• Experience in security/intelligence/operational systems
• Big Data Frameworks (Spark / Flink) Working with
• Experience in video, image, free text and time series processing
• Introduction to Indexing (Elastic, OpenSearch) and Search Engines
• Experience supporting AI/ML models and Feature Stores - • Technology leadership or mentoring for engineers
• Experience working in isolated environments (On-Prem / Air-gapped)