DigiNews

Summary

This data study from Sitedex analyzes the declared robots.txt rules of the top 10,000 sites to understand how training crawlers and answer-time bots are treated. It finds a 2023 spike in GPTBot-related rules and shows that while many sites blocked training crawlers, explicit allowances or mentions for answer-time bots are comparatively rare, suggesting evolving monetization dynamics and openness around live AI responses.