Recreating Minecraft is Not a Benchmark
Summary
Recreating Minecraft is Not a Benchmark argues that popular 'demo benchmarks' (like recreating Minecraft in a prompt) are poor measures of AI capability because they are easy to optimize for and never change. The piece suggests holdout evals and dynamic tests as more meaningful indicators, while acknowledging that demo benchmarks can drive engagement and content but should not define true capability.