TabPFN and TabICL against tuned XGBoost: the model that does not train won on fourteen tables out of fourteen
Summary
Efrain Garay analyzes TabPFN and TabICL against tuned XGBoost across fourteen datasets from the Grinsztajn benchmark. The article presents that a tabular foundation model can predict without training on the target table and, in these tests, consistently outperforms tuned boosting on AUC, though results vary by dataset and width of tables. It also covers licensing hurdles (TabPFN) and the practical trade-offs of using tabular foundation models versus traditional trees.