Web Scraper Developer

Summary

We are seeking a talented and detail-oriented Web Scraper Developer with strong expertise in Python and Selenium. The ideal candidate will have a deep understanding of web scraping methodologies, experience with automation, and the ability to handle complex data extraction requirements from various websites. You’ll work with our data team to extract, clean, and maintain structured data to support our analytical and operational needs.

Key Responsibilities

• Develop, deploy, and maintain web scrapers using Python and Selenium.
• Build automated solutions for large-scale data collection from diverse web sources, ensuring data integrity and efficiency.
• Implement error handling, retries, and other techniques to ensure scraper reliability.
• Collaborate with the data engineering team to integrate web scraping solutions into our data pipelines.
• Optimize scraping scripts for performance, including load balancing and efficient multi-threading.
• Troubleshoot and resolve scraping issues promptly to ensure data freshness and accuracy.
  
  

• Proficiency in Python, particularly in libraries like BeautifulSoup, Scrapy, and Requests.
• Strong experience with Selenium for browser automation.
• Familiarity with HTML, CSS, and web technologies to identify data within web pages.
• Experience with handling dynamic content, cookies, and session management.
• Knowledge of data storage options such as databases (MySQL, MongoDB) or data files (CSV, JSON).
• Strong problem-solving skills and attention to detail.
  
  

• Familiarity with other web scraping tools and frameworks (e.g., Scrapy, Puppeteer).
• Experience with cloud services (AWS, Azure) for large-scale scraping deployments.
• Knowledge of data cleaning and data transformation techniques.
• Understanding of multithreading and concurrency for faster scraping.
• Experience with Git version control.
  
  

• Bachelor’s degree in Computer Science, Data Science, or related field, or equivalent experience.
• 2-5 years of experience in web scraping or a related field.