Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch
Summary
The arXiv paper analyzes whether LLMs can outperform classical hyperparameter optimization algorithms within a fixed compute budget, using an autoresearch testbed. It finds CMA-ES and TPE outperform LLM agents in most settings, and that allowing LLMs to edit code narrows the gap but does not close it. A hybrid approach called Centaur, which shares CMA-ES state with an LLM, achieves the best performance, indicating LLMs are most effective as a complement to classical optimizers. The work also provides open-source code for reproducibility.