DigiNews

Tech Watch by Johan Denoyer

← Back to articles

Creepy crawlies

Quality: 9/10 Relevance: 9/10

Summary

The article analyzes how AI crawlers drain kernel.org resources by rendering commits as HTML for training data. It documents the evolution of anti-scraping measures (IP bans, AS bans, and residential proxies) and notes that legitimate traffic is a small fraction of requests, prompting reductions in crawlable URLs and additional access friction.

🚀 Service construit par Johan Denoyer