How do programming languages impact token efficiency and correctness?
Summary
This piece analyzes how programming languages may influence LLM token costs and correctness, drawing on multiple evals (Zstd, Pandoc) and literature to argue that dynamic vs static typing does not universally determine efficiency. It highlights the need for multiple tasks to draw reliable conclusions, notes correlations between language popularity and outcomes, and discusses evaluation design flaws and the risk of cheating in held-out tests.