PTaRL: Prototype-based Tabular Representation Learning via Space Calibration

TL;DR

This work tackles the instability of deep tabular models caused by representation entanglement and localization. It introduces PTaRL, a prototype-based representation learning framework that constructs a Prototype-based Projection Space ($P$-Space) from global prototypes and projects backbone representations into this space, guided by Optimal Transport to preserve global data structure. Two constraints—Coordinates Diversifying and Prototype Matrix Orthogonalization—promote disentangled representations and independent prototypes. Across multiple datasets and deep tabular models, PTaRL consistently improves predictive performance with statistical significance, and its model-agnostic design enables broad applicability in tabular learning.

Abstract

Tabular data have been playing a mostly important role in diverse real-world fields, such as healthcare, engineering, finance, etc. With the recent success of deep learning, many tabular machine learning (ML) methods based on deep networks (e.g., Transformer, ResNet) have achieved competitive performance on tabular benchmarks. However, existing deep tabular ML methods suffer from the representation entanglement and localization, which largely hinders their prediction performance and leads to performance inconsistency on tabular tasks. To overcome these problems, we explore a novel direction of applying prototype learning for tabular ML and propose a prototype-based tabular representation learning framework, PTaRL, for tabular prediction tasks. The core idea of PTaRL is to construct prototype-based projection space (P-Space) and learn the disentangled representation around global data prototypes. Specifically, PTaRL mainly involves two stages: (i) Prototype Generation, that constructs global prototypes as the basis vectors of P-Space for representation, and (ii) Prototype Projection, that projects the data samples into P-Space and keeps the core global data information via Optimal Transport. Then, to further acquire the disentangled representations, we constrain PTaRL with two strategies: (i) to diversify the coordinates towards global prototypes of different representations within P-Space, we bring up a diversification constraint for representation calibration; (ii) to avoid prototype entanglement in P-Space, we introduce a matrix orthogonalization constraint to ensure the independence of global prototypes. Finally, we conduct extensive experiments in PTaRL coupled with state-of-the-art deep tabular ML models on various tabular benchmarks and the results have shown our consistent superiority.

Figures (6)

• Figure 1: The visualization of representations of deep network w/o and w/ PTaRL with varying model layer depths.
• Figure 2: The PTaRL framework. The original representation of each sample by backbone would be pushed forward to the corresponding projection representation by minimizing the Optimal Transport Distance. The two sentences "coordinates diversifying" and "prototypes orthogonalization" correspond to two constraints for representation disentanglement.
• Figure 3: Visualization of learned representations of deep tabular models w/ and w/o PTaRL.
• Figure 4: P-Space coordinates diversifying visualization of FT-Transformer on HI w/o and w/ Coordinates Diversifying Constraint (D). The first column and second column correspond to the average coordinates values of two different categories, the third column represents the difference of the two categories.
• Figure 5: Global prototypes orthogonalization visualization of MLP on two different tasks.