FORESTLLM: Large Language Models Make Random Forest Great on Few-shot Tabular Learning

TL;DR

ForestLLM reframes few-shot tabular learning by employing a large language model as an offline model designer rather than an online oracle. It replaces traditional, unstable impurity-based splits with LLM-guided semantic partitions that leverage both labeled and unlabeled data, and replaces noisy leaf estimates with in-context, rule-based leaf predictions. The approach preserves the efficiency and interpretability of classical forests while achieving state-of-the-art performance across classification and regression tasks in 4–48 shot settings, including datasets unseen during LLM pretraining. By decoupling LLM inference from deployment, ForestLLM demonstrates how foundation models can distill semantic priors into lightweight, robust neuro-symbolic learners suitable for real-world, data-scarce environments.

Abstract

Tabular data high-stakes critical decision-making in domains such as finance, healthcare, and scientific discovery. Yet, learning effectively from tabular data in few-shot settings, where labeled examples are scarce, remains a fundamental challenge. Traditional tree-based methods often falter in these regimes due to their reliance on statistical purity metrics, which become unstable and prone to overfitting with limited supervision. At the same time, direct applications of large language models (LLMs) often overlook its inherent structure, leading to suboptimal performance. To overcome these limitations, we propose FORESTLLM, a novel framework that unifies the structural inductive biases of decision forests with the semantic reasoning capabilities of LLMs. Crucially, FORESTLLM leverages the LLM only during training, treating it as an offline model designer that encodes rich, contextual knowledge into a lightweight, interpretable forest model, eliminating the need for LLM inference at test time. Our method is two-fold. First, we introduce a semantic splitting criterion in which the LLM evaluates candidate partitions based on their coherence over both labeled and unlabeled data, enabling the induction of more robust and generalizable tree structures under few-shot supervision. Second, we propose a one-time in-context inference mechanism for leaf node stabilization, where the LLM distills the decision path and its supporting examples into a concise, deterministic prediction, replacing noisy empirical estimates with semantically informed outputs. Across a diverse suite of few-shot classification and regression benchmarks, FORESTLLM achieves state-of-the-art performance.

Figures (8)

• Figure 1: An overview of our proposed ForestLLM framework for few-shot tabular learning.
• Figure 2: Hyperparameter sensitivity analysis of ForestLLM with respect to tree depth and number of estimators under varying few-shot settings. The top row shows classification results, and the bottom row shows regression results.
• Figure 3: Ablation study of the impact of unlabeled data. (a) Classification performance measured by AUC. (b) Regression performance measured by NRMSE.
• Figure 4: Comparison of the top 5 most frequently split features on the Gallstone dataset.
• Figure 5: Prompt template and function calling schema used by ForestLLM to generate structured and interpretable split conditions. The task_description provides a concise natural language description of the prediction task, drawn from domain knowledge or dataset metadata (e.g., “Given this person’s characteristics, does this person have diabetes?”). The row_wise_serialized_few_shot_examples lists labeled samples under the current node in a row-wise serialized format, serving as few-shot examples to guide the model’s reasoning. The feature_summary summarizes the features of all samples within the current node — including both labeled and unlabeled data — describing feature types, value distributions for categorical features, and statistics for numerical features.