When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Summary
The paper defines benchmark saturation and analyzes 60 AI benchmarks using 14 saturation-related properties, finding that about half exhibit saturation and saturation tends to increase with the age of the benchmark. It further shows that resilience to saturation is influenced by expert-curation rather than the use of public test data, and discusses design choices to extend benchmark longevity.