Big Data Team Lead

#li-hybrid Bigabid is an innovative technology company led by data scientists and engineers devoted to mobile app growth. Our proprietary ad platform is powered by machine learning and is constantly improving its distribution algorithms.

We deliver valuable results and insights for a fast-growing clientele of major app developers using elite programmatic user acquisition and retargeting technologies.

Our state-of-the-art machine learning technology analyzes 50TB of raw data per day to produce millions of ad recommendations in real-time. This data is used to power our machine learning predictions, business-critical metrics, and analytics to power our decision-making.

As a Big Data Team Lead, you will lead a team focused on building scalable, algorithm-heavy data products rather than owning the underlying data infrastructure.

Your team will be responsible for implementing complex business and optimization logic—often incorporating multiple algorithms using Peta-Byte scale data processing (primarily PySpark). A core part of the role is ensuring these data products are explainable, observable, and trusted by Data Science, BI, Product, and business stakeholders.

You will work closely with the Data Infrastructure TL, Data Science, BI, and Product to turn advanced logic into production-grade, transparent, and measurable systems.

Responsibilities:

Lead development of scalable, algorithm-heavy data products using PySpark and Implement and maintain complex, multi-stage data flows with multiple interacting algorithms.
Translate business and algorithmic requirements into clear, testable data logic. Ensuring explainability of outputs through intermediate artifacts, metadata, and documentation & Defining and monitoring; observability signals for data quality, algorithm health, and business impact.
Mentor engineers on reasoning about complex logic, edge cases, and failure modes.

Requirements:

2+ years of experience in technical leadership or leading complex projects.
6+ years of experience as a Data / Backend Engineer in data-intensive systems.
Strong production experience with PySpark.
Excellent SQL skills for complex analysis, validation, and BI-facing data models.
Proven experience implementing complex algorithms and business logic at scale.
Strong understanding of data correctness, edge cases, and validation strategies.

Advantage

Experience with algorithm explainability and data observability.
Background working closely with Data Science / ML teams.
Experience with large-scale Spark batch or streaming pipelines.
AWS-based data platforms experience.

Excerpt:

Lead a team and manage the project milestones to construct an enterprise-grade real-time data store through the online processing of massive amounts of data and hundreds of thousands of events per second.