Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models

TL;DR

Mitra tackles the problem of generalizing tabular foundation models by designing a principled mixture of synthetic priors that combine SCM and TBP sources. It introduces a quantitative framework using the Generalizability Matrix $\\mathbf{G}$ and Performance Vector $\\mathbf{P}$ to assess prior quality, diversity, and distinctiveness, guiding the construction of a diverse mixture. The resulting Mitra TFM achieves state-of-the-art results on classification and regression benchmarks, exhibits strong sample efficiency, and remains effective across both 1D and 2D attention architectures with ensembling and fine-tuning. This work demonstrates that carefully crafted synthetic priors can substantially improve cross-domain tabular generalization while reducing reliance on real labeled data, and it lays groundwork for incorporating additional priors such as Gaussian processes in future scaling.

Abstract

Since the seminal work of TabPFN, research on tabular foundation models (TFMs) based on in-context learning (ICL) has challenged long-standing paradigms in machine learning. Without seeing any real-world data, models pretrained on purely synthetic datasets generalize remarkably well across diverse datasets, often using only a moderate number of in-context examples. This shifts the focus in tabular machine learning from model architecture design to the design of synthetic datasets, or, more precisely, to the prior distributions that generate them. Yet the guiding principles for prior design remain poorly understood. This work marks the first attempt to address the gap. We systematically investigate and identify key properties of synthetic priors that allow pretrained TFMs to generalize well. Based on these insights, we introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world tabular data. Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks, with better sample efficiency.

Figures (25)

• Figure 1: Mixture of priors in Mitra, Generalizability Matrix $\mathbf{G}$ and Performance Vector $\mathbf{P}$. We consider three factors for good priors: (1) the performance of a pretrained TFM using data generated solely from that prior on real tabular data (Point 1 and $\mathbf{P}$); (2) its diversity and (3) distinctiveness when included in a mixture (Points 2 and 3, captured by the diagonal and off-diagonal entries of $\mathbf{G}$). This leads to our mixture comprising SCM, gradient boosting, random forest and decision tree priors.
• Figure 2: Mitra Pipeline: Model agnostic visualized with either 1D or 2D attention.
• Figure 3: Mitra (bagging) shows the best performance compared with TabPFNv2 PHE, AutoGluon best-quality and Mitra (+ef) across 300- to 3600-second budgets on 1-fold TabRepo benchmark.
• Figure 4: Mitra shows significantly better fine-tuning performance across ensemble sizes.
• Figure 5: Model size scaling law. We vary the model size across six configurations (4, 8, 12, 16, 20, and 24 layers), each pretrained on 45 million unique samples. The plot zooms in on pretraining steps beyond 2,000 to provide a clearer view of performance trends. A larger version focusing only on this zoomed-in region is provided in Figure \ref{['fig:model-scaling-law-large']} in Appendix.