Sometimes giving syndication feed readers good errors is a mistake
Summary
This post discusses the problems with overly generic User-Agent strings used by feed readers and crawlers, argues for clearly identifying User-Agents and blocking high-volume crawlers to reduce server load, and notes the context of data collection for LLM training.