Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PET-TURTLE: Deep Unsupervised Support Vector Machines for Imbalanced Data Clusters

Javier Salazar Cavazos

TL;DR

The paper addresses the challenge of imbalanced clusters in unsupervised deep clustering by extending TURTLE with PET-TURTLE, which adds a prior-enforcement term to align the empirical label distribution with a prior that can be uniform or power-law. It further improves hyperplane estimation by replacing full softmax logits with sparsemax, reducing the effective search space. Empirical results on synthetic (CIFAR10-PL, Food101-PL) and real datasets show PET-TURTLE yields higher clustering accuracy, especially under imbalance, while maintaining gains on balanced data. The approach strengthens unsupervised clustering atop foundation-model representations and suggests future work on nonlinear extensions and broader prior choices, with awareness of biases embedded in the learned representations.

Abstract

Foundation vision, audio, and language models enable zero-shot performance on downstream tasks via their latent representations. Recently, unsupervised learning of data group structure with deep learning methods has gained popularity. TURTLE, a state of the art deep clustering algorithm, uncovers data labeling without supervision by alternating label and hyperplane updates, maximizing the hyperplane margin, in a similar fashion to support vector machines (SVMs). However, TURTLE assumes clusters are balanced; when data is imbalanced, it yields non-ideal hyperplanes that cause higher clustering error. We propose PET-TURTLE, which generalizes the cost function to handle imbalanced data distributions by a power law prior. Additionally, by introducing sparse logits in the labeling process, PET-TURTLE optimizes a simpler search space that in turn improves accuracy for balanced datasets. Experiments on synthetic and real data show that PET-TURTLE improves accuracy for imbalanced sources, prevents over-prediction of minority clusters, and enhances overall clustering.

PET-TURTLE: Deep Unsupervised Support Vector Machines for Imbalanced Data Clusters

TL;DR

The paper addresses the challenge of imbalanced clusters in unsupervised deep clustering by extending TURTLE with PET-TURTLE, which adds a prior-enforcement term to align the empirical label distribution with a prior that can be uniform or power-law. It further improves hyperplane estimation by replacing full softmax logits with sparsemax, reducing the effective search space. Empirical results on synthetic (CIFAR10-PL, Food101-PL) and real datasets show PET-TURTLE yields higher clustering accuracy, especially under imbalance, while maintaining gains on balanced data. The approach strengthens unsupervised clustering atop foundation-model representations and suggests future work on nonlinear extensions and broader prior choices, with awareness of biases embedded in the learned representations.

Abstract

Foundation vision, audio, and language models enable zero-shot performance on downstream tasks via their latent representations. Recently, unsupervised learning of data group structure with deep learning methods has gained popularity. TURTLE, a state of the art deep clustering algorithm, uncovers data labeling without supervision by alternating label and hyperplane updates, maximizing the hyperplane margin, in a similar fashion to support vector machines (SVMs). However, TURTLE assumes clusters are balanced; when data is imbalanced, it yields non-ideal hyperplanes that cause higher clustering error. We propose PET-TURTLE, which generalizes the cost function to handle imbalanced data distributions by a power law prior. Additionally, by introducing sparse logits in the labeling process, PET-TURTLE optimizes a simpler search space that in turn improves accuracy for balanced datasets. Experiments on synthetic and real data show that PET-TURTLE improves accuracy for imbalanced sources, prevents over-prediction of minority clusters, and enhances overall clustering.
Paper Structure (10 sections, 9 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 10 sections, 9 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Prior Enforcement Term for Imbalanced Data
  4. Sparse Logits for Hyperplane Estimation
  5. Experiments & Results
  6. Experimental Setup
  7. Synthetic Results
  8. Real Data Results
  9. Conclusion
  10. Impact Statement

Figures (4)

  • Figure 1: A visual illustration of the key idea behind unsupervised support vector machines that alternates updates between labels and hyperplane estimation.
  • Figure 2: A visual comparison of the effects of regularization terms in the TURTLE and PET-TURTLE objective functions.
  • Figure 3: Probability mass functions of the power law distribution at various decay rates $\alpha$ when $C=10$ as used in Table \ref{['tab:imbalance']}.
  • Figure 4: Confusion matrices of the TURTLE and PET-TURTLE methods on the Food101-PL dataset ($C=101$) with fixed decay rate $\alpha = 1.0$.