Data Engineer

Job Summary:

We are seeking a skilled Data Engineer- AI Integration Specialist with experience in working alongside AI teams to design, build, and optimize scalable data pipelines and infrastructure that support machine learning and AI models. The ideal candidate will collaborate with data scientists, machine learning engineers, and AI specialists to ensure that data is collected, processed, and ready for use in AI-driven applications. Your role is crucial in developing data architectures that enhance AI model performance and ensure seamless integration of AI into business operations.

Key Responsibilities:

• Data Pipeline Development: Design, build, and maintain robust and scalable ETL/ELT pipelines to support the ingestion, processing, and storage of large datasets for AI applications.
• Data Infrastructure: Architect and manage cloud-based and on-premises data infrastructure, ensuring high availability, scalability, and security to support AI and machine learning models.
• Data Preparation for AI: Collaborate with AI and machine learning teams to prepare, clean, and organize large volumes of structured and unstructured data for training, testing, and deployment of AI models.
• Real-Time Data Processing: Implement real-time data streaming solutions (e.g., Apache Kafka, Flink) to enable real-time AI applications and decision-making.
• Data Quality Assurance: Develop and implement data validation checks and automated testing procedures to ensure data integrity and accuracy throughout the pipeline.
• Collaboration with AI Teams: Work closely with data scientists, AI specialists, and business stakeholders to understand data requirements and optimize data flows for machine learning and AI algorithms.
• Performance Optimization: Continuously monitor and enhance data pipelines for performance, including query optimization, indexing, and partitioning for AI workloads.
• Database Management: Manage and maintain large-scale relational and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB) as well as data lakes and warehouses (e.g., AWS S3, Snowflake, BigQuery).
• Cloud Integration: Leverage cloud technologies (AWS, Azure, Google Cloud) to build and deploy scalable data storage and processing solutions, ensuring compatibility with AI tools and frameworks.
• Data Security & Compliance: Ensure that data handling processes comply with industry standards, data governance, and regulatory requirements (GDPR, HIPAA, etc.).

Key Qualifications:

Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or related field.

Required Skills and Experience:

• 3+ years of experience in data engineering, building data pipelines, and working with large datasets.
• Hands-on experience collaborating with AI and machine learning teams to support model development.
• Proficiency in programming languages like Python, SQL, Java, or Scala. Experience with ETL/ELT tools such as Apache Airflow, AWS Glue, Talend, or Informatica.
• Expertise in data streaming technologies like Apache Kafka, Kinesis, or Flink.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud and data storage solutions (e.g., Redshift, BigQuery, Snowflake).
• Experience with big data tools and frameworks (e.g., Hadoop, Spark, Hive).
• Solid understanding of database technologies (SQL and NoSQL) and data warehousing concepts.

Analytical Skills: Strong problem-solving abilities and understanding of how data impacts machine learning and AI models.

Collaboration: Ability to work cross-functionally with AI teams, product managers, and business stakeholders to align on data requirements.

Preferred Qualifications:

• Experience with MLOps tools and practices for automating and scaling machine learning models.
• Knowledge of AI-specific data frameworks and formats (e.g., TensorFlow Data, PyTorch DataLoader).
• Familiarity with data governance, lineage, and cataloging tools (e.g., Apache Atlas, Alation).