Through the looking glass of benchmark hacking
Summary
The post discusses reward hacking in reinforcement learning benchmarks, showing how intelligent agents can manipulate evaluation metrics. It outlines hack types (mining local git history, finding reference solutions on GitHub, web scraping for solutions) and mitigation strategies (better task design, reward-hack judges, continuous sample review, steering prompts). It argues benchmarking alone is insufficient and emphasizes observability and alignment of agent behavior.