Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Data

Patryk Marszałek, Tomasz Kuśmierczyk, Witold Wydmański, Jacek Tabor, Marek Śmieja

TL;DR

ZEUS addresses the challenge of clustering tabular data by learning zero-shot embeddings via a transformer pre-trained on synthetic datasets with latent clustering structure. In the embedding space, clustering probabilities follow $p_k(x) = \frac{\hat{\pi}_k \exp(-\|z(x)-\hat{c}_k\|^2)}{\sum_j \hat{\pi}_j \exp(-\|z(x)-\hat{c}_j\|^2)}$, enabling effective separation using simple methods like $k$-means without any fine-tuning. The model combines a probabilistic clustering objective, compactness and separation regularizers, and a Prior-Data Fitted Networks (PFN) perspective to approximate Bayesian inference through a learned embedding. Empirically, ZEUS achieves competitive or superior clustering performance across real OpenML data and synthetic datasets, while offering fast inference and reduced hyperparameter tuning. This work enables practical, zero-shot unsupervised clustering in tabular domains and provides a principled framework for synthetic-data priors and transformer-based embeddings.

Abstract

Clustering tabular data remains a significant open challenge in data analysis and machine learning. Unlike for image data, similarity between tabular records often varies across datasets, making the definition of clusters highly dataset-dependent. Furthermore, the absence of supervised signals complicates hyperparameter tuning in deep learning clustering methods, frequently resulting in unstable performance. To address these issues and reduce the need for per-dataset tuning, we adopt an emerging approach in deep learning: zero-shot learning. We propose ZEUS, a self-contained model capable of clustering new datasets without any additional training or fine-tuning. It operates by decomposing complex datasets into meaningful components that can then be clustered effectively. Thanks to pre-training on synthetic datasets generated from a latent-variable prior, it generalizes across various datasets without requiring user intervention. To the best of our knowledge, ZEUS is the first zero-shot method capable of generating embeddings for tabular data in a fully unsupervised manner. Experimental results demonstrate that it performs on par with or better than traditional clustering algorithms and recent deep learning-based methods, while being significantly faster and more user-friendly.

ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Data

TL;DR

ZEUS addresses the challenge of clustering tabular data by learning zero-shot embeddings via a transformer pre-trained on synthetic datasets with latent clustering structure. In the embedding space, clustering probabilities follow , enabling effective separation using simple methods like -means without any fine-tuning. The model combines a probabilistic clustering objective, compactness and separation regularizers, and a Prior-Data Fitted Networks (PFN) perspective to approximate Bayesian inference through a learned embedding. Empirically, ZEUS achieves competitive or superior clustering performance across real OpenML data and synthetic datasets, while offering fast inference and reduced hyperparameter tuning. This work enables practical, zero-shot unsupervised clustering in tabular domains and provides a principled framework for synthetic-data priors and transformer-based embeddings.

Abstract

Clustering tabular data remains a significant open challenge in data analysis and machine learning. Unlike for image data, similarity between tabular records often varies across datasets, making the definition of clusters highly dataset-dependent. Furthermore, the absence of supervised signals complicates hyperparameter tuning in deep learning clustering methods, frequently resulting in unstable performance. To address these issues and reduce the need for per-dataset tuning, we adopt an emerging approach in deep learning: zero-shot learning. We propose ZEUS, a self-contained model capable of clustering new datasets without any additional training or fine-tuning. It operates by decomposing complex datasets into meaningful components that can then be clustered effectively. Thanks to pre-training on synthetic datasets generated from a latent-variable prior, it generalizes across various datasets without requiring user intervention. To the best of our knowledge, ZEUS is the first zero-shot method capable of generating embeddings for tabular data in a fully unsupervised manner. Experimental results demonstrate that it performs on par with or better than traditional clustering algorithms and recent deep learning-based methods, while being significantly faster and more user-friendly.
Paper Structure (40 sections, 1 theorem, 10 equations, 4 figures, 25 tables)

This paper contains 40 sections, 1 theorem, 10 equations, 4 figures, 25 tables.

Table of Contents

  1. Introduction
  2. Pre-training representation for clustering
  3. ZEUS method
  4. Zero-shot pre-training.
  5. Representation learning for probabilistic clustering.
  6. Inference.
  7. Regularization.
  8. Prior data for pre-training ZEUS
  9. Relation to Bayesian learning and Prior-Data Fitted Networks
  10. Experiments
  11. Experimental setup
  12. How effective is ZEUS for clustering?
  13. Are ZEUS's predictions calibrated?
  14. How helpful is regularization for ZEUS?
  15. What data-generating prior is optimal?
  16. ...and 25 more sections

Key Result

Theorem 1

Let $P = \sum_{i=1}^k \pi_i P_i$ be a mixture of distributions on $\mathbb{R}^d$, where each $P_i$ is a probability distribution, and $\pi_i > 0$, $\sum_{i=1}^k \pi_i = 1$. Then there exists a mixture of Gaussians $Q = \sum_{i=1}^k \pi_i \mathcal{N}(\mu_i, \Sigma_i)$ and a neural network $F$ such th

Figures (4)

  • Figure 1: Schematic characterization of ZEUS: (left) synthetic datasets generation; (middle) pre-training on datasets with known labels; (right) deployment of a frozen model for real-world tasks.
  • Figure 2: t-SNE visualization for a sample synthetic dataset. The representation from ZEUS (right panel) significantly improves consistency with ground-truth classes and reveals a clearer data structure.
  • Figure 3: Clustering time vs. input size.
  • Figure 4: Visualization of pre-training process

Theorems & Definitions (5)

  • Remark 2.1
  • Remark 2.2
  • Remark 2.3
  • Theorem 1: Univeral Approximation Theorem for Mixture Distributions
  • proof