Scraping Engineer

You are here to turn clarity into impact.

dig is a fast-growing Social Video Intelligence startup backed by top investors, giving brands the power to finally see and control what’s happening to them on social video.

We are seeking a specialist who lives in the Network tab of DevTools. You will be responsible for designing, building, and maintaining the infrastructure that powers our data ingestion, ensuring our data pipeline delivers high success rates at scale.

What you will do:

• Reverse Engineering: Dissect complex websites and mobile APIs (Android/iOS) to find hidden endpoints and efficiently extract data without rendering.
• Infrastructure & Scale: Build and maintain scalable scrapers using Python that handle millions of requests daily.
• Bypass Protections: Develop strategies to overcome anti-bot measures, including TLS fingerprinting, CAPTCHA solving, and header optimization.
• Proxy Management: Orchestrate smart proxy rotation strategies.
• Data Quality: Ensure the integrity and structure of harvested data, implementing automated validation.
• Tooling: Maintain our internal scraping framework.

Requirements:

• Python Expertise: Strong proficiency in Python.
• Network Fundamentals: Mastery of HTTP/S, TLS, cookies, and headers. You know exactly what happens in a TLS handshake and how to mimic it.
• Database Knowledge: Experience with data storage (PostgreSQL, MongoDB, Redis) and message queues (Kafka, RabbitMQ, SQS).

Why dig

We are a 50 person team in Tel Aviv revolutionizing the social listening category with urgency, clarity, and heart. A place with energy, humor, belonging, and real ownership. A product with meaning. A team you will want to show up for. If you want to help define a category and shape how brands navigate the Video First world, we want to meet you.