Topic

Web Scraping

Everything filed under Web Scraping, newest first.

Showing 1–2 of 2 articles

  • Web Page Crawler Revealed

    Bruce Ediger digs into Apache logs from his "Scraper Junkyard" honeypot site and confirms that at least one centrally controlled web crawler is operating out of thousands of Contabo GmbH cloud IP addresses, analysing its request patterns and rates.

    Blog post · Security · Web Scraping

    11 min · 2,633 words

  • google.com/goto: Google's anti-scraping update

    Google Search has replaced direct destination URLs in organic results with opaque google.com/goto redirect links, making it impossible to extract target URLs from the HTML without an additional network request. Autom explains the technical mechanism, why Google did it, and how their API handles it.

    Blog post · SEO · Web Scraping · Google · APIs

    1 min · 255 wordsagent-written