Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective

TL;DR

This work tackles the challenge of transferring knowledge across heterogeneous tabular datasets by introducing TabPTM, a pre-training framework that uses a neighborhood-based meta-representation. Each instance is encoded by its distances to the $K$ nearest neighbors and their labels, producing a fixed-dimensional input regardless of original feature spaces, and a shared top-layer model $T_{\Theta}$ learns from these representations across many datasets. The model can be applied directly to unseen datasets or fine-tuned with minimal parameter updates, achieving strong classification and regression performance, especially in few-shot settings, and often surpassing existing pre-trained tabular models. The results on 101 datasets demonstrate the potential of neighborhood-informed meta-representations to serve as a universal vocabulary for tabular transfer learning, enabling scalable, cross-dataset generalization without extensive task-specific tuning.

Abstract

Pre-training is prevalent in deep learning for vision and text data, leveraging knowledge from other datasets to enhance downstream tasks. However, for tabular data, the inherent heterogeneity in attribute and label spaces across datasets complicates the learning of shareable knowledge. We propose Tabular data Pre-Training via Meta-representation (TabPTM), aiming to pre-train a general tabular model over diverse datasets. The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels. This ''meta-representation'' transforms heterogeneous tasks into homogeneous local prediction problems, enabling the model to infer labels (or scores for each label) based on neighborhood information. As a result, the pre-trained TabPTM can be applied directly to new datasets, regardless of their diverse attributes and labels, without further fine-tuning. Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.

Figures (7)

• Figure 1: An illustration of TabPTM vs. other training strategies on tabular data. Left: the vanilla training and prediction pipeline, where tabular models are trained on each dataset separately. Middle: pre-training of a joint model on top of the learned dataset-specific token representations (of each attribute), in which the representation must be re-trained for downstream datasets. Right: TabPTM unifies heterogeneous tabular datasets via the meta-representation using neighborhood information, allowing pre-training a shareable model to predict labels based on such information, even without fine-tuning it on downstream datasets.
• Figure 2: An illustration of the Meta-Representation (MR) for classification. MR transforms heterogeneous tabular datasets with different dimensions into a homogeneous form. A dataset has a set of $K$-dimension MRs, one for each class, which can be used to derive the prediction scores for different classes. We pre-train a joint model on the MRs of multiple datasets and extend its generalization ability to downstream datasets. The right figure shows the MR of an instance (a row in the original tabular format), encoding distances from the instance to its nearest neighbors of each class, characterizing the class membership patterns.
• Figure 3: Pilot study of Meta-Representation (MR) over two datasets "breast-cancer-wisc" (binary, denoted by "+") and "dermatology" (multi-class, denoted by "$\circ$"), using t-SNE van2008visualizing. We use colors to distinguish classes. (a) and (b) display the characteristics of each dataset. In (c), we homogenize the datasets using MR, with red indicating the MR for the target class ($\phi_{y_i}({\bm{x}}_i)$) and blue for non-target classes ($\phi_{c \neq y_i}({\bm{x}}_i)$). MR effectively unifies datasets into a joint representation space where their classifications can be implemented by the dotted line.
• Figure 4: Performance of different methods on few-shot tasks across 65 datasets. For each downstream dataset, $\{4,8,16,32,64\}$ examples per class (shot) are randomly sampled as the training set. TabPTM outperforms other methods overall. The complete results for all few-shot tasks are presented in \ref{['tab:fewshot_main_whole']} and \ref{['tab:fewshot_tinybench_whole']}. "Overall" averages the ranks among all shots.
• Figure 5: Performance comparison across 18 classification and 18 regression datasets in the full-shot setting, where the entire training set is used. The pre-trained TabPTM, trained on 18 non-overlapping datasets, outperforms all other methods and achieves the highest average rank. The complete results are reported in \ref{['tab:reg_main']} and \ref{['tab:cls_main']}.