Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard

TL;DR

The paper tackles the challenge of achieving high accuracy on tabular prediction across diverse datasets by leveraging large language models as instance-aware ensemble agents. It constructs a non-semantic tabular context from nearest neighbors and external model predictions, and applies Chain of Tabular Thoughts (CoT$^2$) to guide multi-step, interpretable reasoning for final predictions. Empirical results on TinyBench2 show CoT$^2$ outperforms well-tuned baselines and standard ensembling, while employing selective LLM inference to cut costs. The work advances privacy-preserving, context-driven usage of LLMs in tabular ML, with practical impact for ensemble design in real-world datasets.

Abstract

Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications. The performance of tabular models--such as gradient boosted decision trees and neural networks--can vary significantly across datasets due to differences in feature distributions and task characteristics. Achieving top performance on each dataset often requires specialized expert knowledge. To address this variability, practitioners often aggregate the predictions of multiple models. However, conventional aggregation strategies typically rely on static combination rules and lack instance-level adaptability. In this work, we propose an in-context ensemble framework for tabular prediction that leverages large language models (LLMs) to perform dynamic, instance-specific integration of external model predictions. Without access to raw tabular features or semantic information, our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models. Within this enriched context, we introduce Chain of Tabular Thoughts (CoT$^2$), a prompting strategy that guides LLMs through multi-step, interpretable reasoning, making still further progress toward expert-level decision-making. Experimental results show that our method outperforms well-tuned baselines and standard ensemble techniques across a wide range of tabular datasets.

Figures (18)

• Figure 1: CoT$^2$ utilizes the expert knowledge of LLMs to create an intelligent ensemble of tabular models, making still further progress.
• Figure 2: An example of a binary classification task using the tabular context and Chain of Tabular Thoughts (CoT$^2$). We construct the tabular context based on the combination of neighbors and external model predictions. We design reasoning steps by learning from the thought processes of leaderboard experts. Experts typically first filter models and neighbors, then make predictions by aggregating the external models' predictions for the neighbors and target instances. The tabular context and CoT$^2$ are both provided as a prompt to the LLMs. \ref{['fig:prompt']} shows an example.
• Figure 3: Critical difference diagram based on the Wilcoxon-Holm test with a significance level of 0.05, used to assess pairwise significance of methods on 30 classification datasets in TinyBench2. Blue-colored methods represent the models included in the external model set. The method names in the diagram are abbreviated; the mapping from abbreviations to full names can be found in Ye2024Closer and \ref{['append:datasets']}.
• Figure 4: Impact of external model set size and quality on the performance of CoT$^2$.
• Figure 5: Performance of CoT$^2$ under different numbers of neighbors $k$ used in the context.