On Learning Representations for Tabular Data Distillation

TL;DR

The paper tackles tabular data distillation by addressing feature heterogeneity and non-differentiable downstream models through TDColER, a column-embedding based representation learning framework that enables distillation in a learned latent space. It combines column embeddings with encoders (FFN, Transformer, GNN) and a reconstruction-based objective, optionally augmented with supervised fine-tuning, to produce compact, information-rich representations that improve downstream performance when paired with off-the-shelf distillation schemes. The authors introduce TDBench, a comprehensive benchmark across 23 OpenML datasets, 11 distillation schemes, 3 autoencoder architectures, and 7 downstream classifiers, reporting up to 0.5–143% gains in distilled data quality and providing insights such as the superiority of latent-space clustering (notably KM with TF-SFT) and the robustness of clustering under class imbalance. The work demonstrates that learning latent representations for tabular data distillation substantially boosts cross-architecture performance, proposes a practical, model-agnostic pipeline, and offers a reproducible benchmark to guide future research in tabular data distillation with real-world impact on storage, privacy, and compute efficiency.

Abstract

Dataset distillation generates a small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality. We study tabular data distillation, which brings in novel challenges such as the inherent feature heterogeneity and the common use of non-differentiable learning models (such as decision tree ensembles and nearest-neighbor predictors). To mitigate these challenges, we present $\texttt{TDColER}$, a tabular data distillation framework via column embeddings-based representation learning. To evaluate this framework, we also present a tabular data distillation benchmark, ${\sf \small TDBench}$. Based on an elaborate evaluation on ${\sf \small TDBench}$, resulting in 226,890 distilled datasets and 548,880 models trained on them, we demonstrate that $\texttt{TDColER}$ is able to boost the distilled data quality of off-the-shelf distillation schemes by 0.5-143% across 7 different tabular learning models.

Figures (22)

• Figure 1: Overview of TDBench. The benchmarking suite allows for flexible choice of datasets, distillation schemes, and downstream models that enables for modular evaluation of any new distillation method.
• Figure 1: Average rank and median relative regret of distillation pipelines that use the latent space of different encoder architectures evaluated at IPC=10, grouped over all datasets and classifiers.
• Figure 2: Proposed approach -- TDColER: Tabular Distillation Via Column Embeddings based Representation Learning
• Figure 3: Change in relative regret of downstream classifiers when trained on distilled data over IPC $\in [10, 100]$, aggregated over datasets and encoder architectures. Lower is better. Each column corresponds to a downstream classifier, and each row represents a representation scheme -- original, encoded (Enc.), and reconstructed (Rec.). Data distilled by clustering methods (AG, KM) in the encoded space show the best performance for all classifiers. In many cases, using the encoded representation as the final output yields a performance comparable to using the original representation. \ref{['fig:apdx:dm-per-clf-per-dm']} shows a more detailed version of this plot that includes FTTransformer and ResNet.
• Figure 3: The best performers of each dataset are classifiers ranked by their appearance count at the top $3$ of each comparison at IPC=10. $k$-means stands out as the strongest performer in combination with a supervised-fine-tuned transformer encoder.

Theorems & Definitions (1)

• Remark 1