Web Scraping

webrecorder/browsertrix-crawler: Run a high-fidelity browser-based crawler in a single Docker container

• crawlee
  • Can use Headless browsers

Web Scraping Tools

• Dynamic Sites
  • Selenium
  • nodejs
    • Puppeteer
• Scraping Static Sites
  • Software/List/curl
  • wget
  • Python
    • Python requests package
    • pywebcopy
• Parsing content
  • Beautiful Soup
  • pyquery · PyPI
  • sqlite HTML extension

Links

• Web Scraping Browser Extensions

Sources

• Top 5 Open Source Web Scraping Frameworks and Libraries - Datahut

Backlinks

• Web Crawler Software
• Archive Software