Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification

Yanbiao Ma, Licheng Jiao, Fang Liu, Lingling Li, Shuyuan Yang, Xu Liu

TL;DR

The paper tackles the brittleness of confidence-based pseudo-labeling in semi-supervised learning by introducing Hierarchical Dynamic Labeling (HDL), an embedding-based labeling method that does not rely on model predictions. HDL leverages image embeddings (eg., CLIP) and a hierarchical, adaptive process to relabel unlabeled data and continue training, with an explicit mechanism for adaptive $k$ selection. It formalizes $(k,\delta_k)$ label clusterability and demonstrates that HDL consistently improves a range of semi-supervised methods on both class-balanced and long-tailed datasets, often surpassing traditional kNN-based labeling approaches. The work suggests a data-centric shift in pseudo-label generation, enabling more robust performance across diverse domains and providing a versatile data-processing module for semi-supervised pipelines.

Abstract

In semi-supervised learning, methods that rely on confidence learning to generate pseudo-labels have been widely proposed. However, increasing research finds that when faced with noisy and biased data, the model's representation network is more reliable than the classification network. Additionally, label generation methods based on model predictions often show poor adaptability across different datasets, necessitating customization of the classification network. Therefore, we propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels. We also introduce an adaptive method for selecting hyperparameters in HDL, enhancing its versatility. Moreover, HDL can be combined with general image encoders (e.g., CLIP) to serve as a fundamental data processing module. We extract embeddings from datasets with class-balanced and long-tailed distributions using pre-trained semi-supervised models. Subsequently, samples are re-labeled using HDL, and the re-labeled samples are used to further train the semi-supervised models. Experiments demonstrate improved model performance, validating the motivation that representation networks are more reliable than classifiers or predictors. Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.

Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification

TL;DR

The paper tackles the brittleness of confidence-based pseudo-labeling in semi-supervised learning by introducing Hierarchical Dynamic Labeling (HDL), an embedding-based labeling method that does not rely on model predictions. HDL leverages image embeddings (eg., CLIP) and a hierarchical, adaptive process to relabel unlabeled data and continue training, with an explicit mechanism for adaptive selection. It formalizes label clusterability and demonstrates that HDL consistently improves a range of semi-supervised methods on both class-balanced and long-tailed datasets, often surpassing traditional kNN-based labeling approaches. The work suggests a data-centric shift in pseudo-label generation, enabling more robust performance across diverse domains and providing a versatile data-processing module for semi-supervised pipelines.

Abstract

In semi-supervised learning, methods that rely on confidence learning to generate pseudo-labels have been widely proposed. However, increasing research finds that when faced with noisy and biased data, the model's representation network is more reliable than the classification network. Additionally, label generation methods based on model predictions often show poor adaptability across different datasets, necessitating customization of the classification network. Therefore, we propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels. We also introduce an adaptive method for selecting hyperparameters in HDL, enhancing its versatility. Moreover, HDL can be combined with general image encoders (e.g., CLIP) to serve as a fundamental data processing module. We extract embeddings from datasets with class-balanced and long-tailed distributions using pre-trained semi-supervised models. Subsequently, samples are re-labeled using HDL, and the re-labeled samples are used to further train the semi-supervised models. Experiments demonstrate improved model performance, validating the motivation that representation networks are more reliable than classifiers or predictors. Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
Paper Structure (19 sections, 3 equations, 3 figures, 3 tables)

This paper contains 19 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Task and Symbol Definition
  5. Label Clusterability
  6. Automatic classification using embedding
  7. Intuition for image labeling relying on $(k,\delta _k)$ label clustrability
  8. Unlabeled embeddings can promote labeling
  9. Hierarchical Dynamic Labeling (HDL)
  10. Adaptive selection and analysis of $k$
  11. Experiments
  12. Experimental objectives and program design
  13. Datasets and Implementation Details
  14. Setting of the hyperparameter $k$
  15. Results on Class-Balanced Datasets
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An example of a two-tier tree structure used for label searching. Blue boxes indicate the absence of a label, while yellow boxes represent uncertainty regarding the existence of a label.
  • Figure 2: The probability $\mu_3$ that clustrability holds on CIFAR-10, CIFAR-100 and STL-10 for $k = 3$.
  • Figure 3: Under different values of $k$, the means and standard deviations of the probabilities that the embedding sets for the four image datasets satisfy clusterability. Please refer to the blue vertical axis for the magnitude of the mean and the red vertical axis for the magnitude of the standard deviation.

Theorems & Definitions (1)

  • Definition 1: $(k,\delta_k)$ Label Clusterability