TIS-100 Bench: LLMs Push the Pareto Frontier in Programming Puzzle Games
Summary
The article presents a TIS-100 benchmark evaluating 16 LLMs on Zachtronics puzzles, highlighting frontier points and Pareto tradeoffs among cycles, nodes, and instructions. It compares model efficiency, cost, and generalization through a rigorous testing regime (100,000 passing tests) and provides downloadable datasets, code, and a Pareto explorer for deeper analysis.