Data Engineering Team Leader

We're seeking a skilled Data Engineer to work on Palantir's data infrastructure and analytics platforms. The ideal candidate will combine strong technical skills with the ability to design, implement, and optimize data pipelines for complex data systems.

• Design, build, and maintain a robust DevOps and data infrastructure from scratch.
• Lead and mentor a team in managing cloud-based infrastructure across AWS and Azure.
• Implement Kubernetes, CI/CD pipelines, and scalable deployment strategies.
• Ensure high system reliability, security, and performance for real-time applications.
• Work closely with Data Scientists and Backend Engineers to build and optimize data pipelines.
• Start with 90% hands-on work, gradually transitioning into a leadership role.
• Design, build, and maintain scalable data processing systems
• Develop ETL processes and data pipelines using PySpark and Pandas
• Collaborate with data scientists and software engineers to implement data-driven solutions
• Optimize queries and improve data model performance
• Ensure data quality, reliability, and accessibility
• Participate in code reviews and documentation
  
  

• 5+ years of hands-on experience in DevOps and Data Engineering.
• 3+ years of experience leading a DevOps or Data Engineering team.
• Strong experience with AWS, Azure, Kubernetes, and CI/CD pipelines.
• Ability to architect, implement, and optimize cloud-based and on-prem solutions.
• Strong problem-solving skills in real-time and high-scale environments.
• Experience with PySpark for large-scale data processing
• Expertise in Pandas for data manipulation and analysis
• Experience with SQL
• Knowledge of data modeling and data warehouse concepts
  
  

• Experience with Kafka, Airflow, Snowflake, or Redshift.
• Prior experience in building data infrastructure from scratch.
• Security and compliance knowledge (ISO, NIST, encryption best practices).
• Ability to work in an Agile environment with cross-functional teams.
• Experience working with language models and NLP technologies
• Knowledge of DataOps principles and practices
• Experience with Elasticsearch for search and analytics
• Familiarity with cloud platforms (AWS, GCP, or Azure)
• Experience with streaming data technologies
• Knowledge of Palantir's products and platforms