Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statsformer: Validated Ensemble Learning with LLM-Derived Semantic Priors

Erica Zhang, Naomi Sagan, Danny Tse, Fangzhao Zhang, Mert Pilanci, Jose Blanchet

TL;DR

Statsformer presents a principled approach to integrating LLM-derived semantic priors into supervised learning through a guardrailed ensemble. By injecting priors with monotone transformations and validating their utility via out-of-fold stacking, it achieves oracle-like guarantees relative to convex combinations of base learners while automatically downweighting unreliable priors. Empirical results across diverse high-dimensional tabular datasets show consistent gains from informative priors and robustness to priors of varying quality. The framework is scalable, model-agnostic, and data-corroborated, offering a practical path to leveraging foundation-model knowledge in standard predictive tasks.

Abstract

We introduce Statsformer, a principled framework for integrating large language model (LLM)-derived knowledge into supervised statistical learning. Existing approaches are limited in adaptability and scope: they either inject LLM guidance as an unvalidated heuristic, which is sensitive to LLM hallucination, or embed semantic information within a single fixed learner. Statsformer overcomes both limitations through a guardrailed ensemble architecture. We embed LLM-derived feature priors within an ensemble of linear and nonlinear learners, adaptively calibrating their influence via cross-validation. This design yields a flexible system with an oracle-style guarantee that it performs no worse than any convex combination of its in-library base learners, up to statistical error. Empirically, informative priors yield consistent performance improvements, while uninformative or misspecified LLM guidance is automatically downweighted, mitigating the impact of hallucinations across a diverse range of prediction tasks.

Statsformer: Validated Ensemble Learning with LLM-Derived Semantic Priors

TL;DR

Statsformer presents a principled approach to integrating LLM-derived semantic priors into supervised learning through a guardrailed ensemble. By injecting priors with monotone transformations and validating their utility via out-of-fold stacking, it achieves oracle-like guarantees relative to convex combinations of base learners while automatically downweighting unreliable priors. Empirical results across diverse high-dimensional tabular datasets show consistent gains from informative priors and robustness to priors of varying quality. The framework is scalable, model-agnostic, and data-corroborated, offering a practical path to leveraging foundation-model knowledge in standard predictive tasks.

Abstract

We introduce Statsformer, a principled framework for integrating large language model (LLM)-derived knowledge into supervised statistical learning. Existing approaches are limited in adaptability and scope: they either inject LLM guidance as an unvalidated heuristic, which is sensitive to LLM hallucination, or embed semantic information within a single fixed learner. Statsformer overcomes both limitations through a guardrailed ensemble architecture. We embed LLM-derived feature priors within an ensemble of linear and nonlinear learners, adaptively calibrating their influence via cross-validation. This design yields a flexible system with an oracle-style guarantee that it performs no worse than any convex combination of its in-library base learners, up to statistical error. Empirically, informative priors yield consistent performance improvements, while uninformative or misspecified LLM guidance is automatically downweighted, mitigating the impact of hallucinations across a diverse range of prediction tasks.
Paper Structure (93 sections, 6 theorems, 98 equations, 16 figures, 5 tables, 1 algorithm)

This paper contains 93 sections, 6 theorems, 98 equations, 16 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Supervised Statistical Learning
  4. Ensemble Learning.
  5. Semantic Priors in Statistical Learning
  6. Methodology
  7. Prior Injection via Monotone Transformations.
  8. Monotone map.
  9. Injection mechanisms.
  10. Prior Validation With Out-Of-Fold (OOF) Stacking
  11. Oracle Guarantees
  12. Experiments
  13. Implementation Details
  14. Datasets and Data Splitting.
  15. Statsformer Implementation.
  16. ...and 78 more sections

Key Result

Theorem 1

Assume $\ell(\cdot, y)$ is convex, $L_\ell$-Lipschitz, and $\ell_{\mathrm{max}} \coloneqq \sup_{y \in \mathcal{Y}} |\ell(0,y)|$ is bounded. Furthermore, assume that all cross-fitted predictors satisfy $\|f_l^{(-k)}(x)\|\le B$ for all $k, l, x$. Then, for any $\delta\in(0,1)$, with probability at lea where $C>0$ is a universal constant.In practice $K$ is fixed and small, so the $K$-dependent factor

Figures (16)

  • Figure 1: Statsformer performance on a variety of datasets, compared to a variety of baseline methods. Note that, due to computational constraints, we only included the AutoML-Agent baseline in Bank Marketing, ETP, and Lung Cancer (see Table \ref{['tab:computation']} in the Appendix for a more detailed computational comparison). For all datasets, we plot either accuracy or AUROC, where higher is better, except Superconductivity, where we plot mean squared error (lower is better). For each training ratio, we plot the mean of the selected metrics 10 different train-test splits (selected via stratified splitting), as well as 95% confidence intervals. Due to the low-sample and imbalanced nature of ETP, we limit the training sizes to be in between $0.3$ and $0.7$ to allow sufficient positive samples in each training and test split.
  • Figure 2: Direct accuracy and AUROC comparison of Statsformer to Statsformer (no prior) for selected datasets. Gains are noticeable across all four examples, and significant for ETP. See Figure \ref{['fig:appdx_us_vs_stacking']} in the Appendix for datasets not shown here.
  • Figure 3: Left: Win ratio of the adversarial-prior Statsformer (pink) and the no-prior Statformer (brown), computed as the percentage of train-test splits where one method performs at least as well as the other. Right: For the methods where Statsformer achieves the lowest win ratios, we plot the corresponding accuracy or AUROC to show that the magnitude of the difference is relatively small.
  • Figure 4: Mean performance improvement of Statsformer over Statsformer (no priors), with prior scores generated by various LLM choices. We present more about experimental setting and additional results in Appendix \ref{['model_ablation_append']}. For all datasets, we plot AUROC, where higher is better. Qwen2.5 Instruct (7B) is (arguably) the weakest LLM among all choices, whereas Claude overall performs well.
  • Figure 5: Single-learner study on selected datasets for prior injection into weighted Lasso (using the adelie Python package).
  • ...and 11 more figures

Theorems & Definitions (16)

  • Theorem 1: Oracle Guarantees for Validated Prior Integration
  • proof
  • Corollary 1: Oracle Guarantee for the Refit Statsformer Predictor
  • proof
  • proof
  • proof
  • Lemma 1: Refit gap bound from consistency
  • proof
  • Proposition 1: Consistency of strongly convex ERM
  • proof : Proof sketch
  • ...and 6 more