EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
Summary
EsoLang-Bench evaluates 80 programming problems across five esoteric languages to test genuine reasoning in large language models. Results show a large gap between Python performance (~90%) and esoteric tasks (~3.8%), with Whitespace entirely unsolved and tool-augmented, interpreter-backed approaches offering the best gains. The findings highlight limits of prompting alone and the potential of agentic systems with execution feedback for improving code-generation reasoning.