How We Broke Top AI Agent Benchmarks: And What Comes Next
Summary
Berkeley researchers show that eight top AI agent benchmarks can be gamed to achieve near-perfect scores without real task solving. They detail concrete exploits across benchmarks like SWE-bench, Terminal-Bench, WebArena, FieldWorkArena, OSWorld, GAIA, and CAR-bench, demonstrating vulnerabilities such as no isolation between agent and evaluator, answers shipped with tests, eval() on untrusted input, unsanitized LLM judges, weak string matching, flawed evaluation logic, and trusting untrusted code. The piece argues that the reputation of these benchmarks is compromised and presents the Agent-Eval Checklist to fix evaluation pipelines, emphasizing isolation, proper sanitization, adversarial testing, data trace integrity, robust scoring, and secrecy of held-out answers. It also introduces BenchJack, a vulnerability scanner that fuzzes benchmarks and generates executable exploits to reveal weaknesses, and it advocates integrating adversarial robustness into the benchmark lifecycle before publication or updates.