Rethinking Guidance Information to Utilize Unlabeled Samples:A Label Encoding Perspective

TL;DR

This work tackles the challenge of learning with scarce labeled data by rethinking the role of guidance information for unlabeled samples. It introduces Label-Encoding Risk Minimization (LERM), which estimates category-specific label encodings from unlabeled predictions via prediction means and minimizes the divergence to the ground-truth one-hot encodings, thereby achieving both prediction discriminability and diversity. The authors establish theoretical connections between LERM and ERM, as well as between LERM and EntMin, and demonstrate substantial empirical gains across SSL, UDA, SHDA, and even SFDA benchmarks. The approach acts as a versatile plugin that can enhance a wide range of existing methods without requiring domain-specific redesign, offering a practical and principled alternative to EntMin for leveraging unlabeled data. The work also provides extensive analyses, including convergence, diversity under class-imbalance, and parameter sensitivity, supporting the robustness and scalability of LERM.

Abstract

Empirical Risk Minimization (ERM) is fragile in scenarios with insufficient labeled samples. A vanilla extension of ERM to unlabeled samples is Entropy Minimization (EntMin), which employs the soft-labels of unlabeled samples to guide their learning. However, EntMin emphasizes prediction discriminability while neglecting prediction diversity. To alleviate this issue, in this paper, we rethink the guidance information to utilize unlabeled samples. By analyzing the learning objective of ERM, we find that the guidance information for labeled samples in a specific category is the corresponding label encoding. Inspired by this finding, we propose a Label-Encoding Risk Minimization (LERM). It first estimates the label encodings through prediction means of unlabeled samples and then aligns them with their corresponding ground-truth label encodings. As a result, the LERM ensures both prediction discriminability and diversity, and it can be integrated into existing methods as a plugin. Theoretically, we analyze the relationships between LERM and ERM as well as EntMin. Empirically, we verify the superiority of the LERM under several label insufficient scenarios. The codes are available at https://github.com/zhangyl660/LERM.

Figures (5)

• Figure 1: Illustrations of the ERM and LERM. Here, different shapes denote distinct categories. In the ERM, we can observe that the six labeled samples are mapped into three label encodings associated with distinct categories. Also, the label encodings of labeled samples remain consistent with those of unlabeled samples. This inspires us to apply those label encodings as guidance information to supervise the learning of unlabeled samples. To this end, we propose the LERM. It first estimates the label encodings through prediction means for unlabeled samples and then aligns them with their corresponding ground-truth label encodings.
• Figure 2: Comparison between LERM and ERM under the SSL setting.
• Figure 3: Empirical evaluation of prediction diversity on the SSL task on CIFAR-10 dataset under the class-imbalanced setting. (a) The ground-truth category distributions of the labeled and unlabeled samples. (b) The predicted category distribution of the unlabeled samples by ERM + EntMin. (c) The predicted category distribution of the unlabeled samples by ERM + LERM.
• Figure 4: Parameter sensitivity analysis on the SHDA tasks of E$\rightarrow$S5 and N$\rightarrow$I.
• Figure 5: t-SNE visualization for the UDA task A$\rightarrow$D on the Office-31 dataset. The red and blue circles represent the source and target features, respectively.

Theorems & Definitions (3)

• Theorem 4.1
• Theorem 4.2
• Theorem 4.3