The Scientific Literature is Poisonous to LLMs
Summary
The article argues that the 21st century scientific literature can hinder LLM training due to mixed quality, misrepresentation, and misconduct across papers. It cites a 2024 multi-institution study showing that removing ArXiv, PhilPapers, and NIH ExPorter from training data improved LLM performance and reduced toxic output, while PubMed removal had mixed effects. It discusses how AI agents lack informal social networks used by scientists to verify work, implying AI for science may need new governance or informal AI-assisted workflows, including surveillance trade-offs.