Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RelationMatch: Matching In-batch Relationships for Semi-supervised Learning

Yifan Zhang, Jingqin Yang, Zhiquan Tan, Yang Yuan

TL;DR

RelationMatch introduces Matrix Cross-Entropy (MCE), a theoretically grounded loss that enforces in-batch relational consistency between weak and strong augmentations in semi-supervised learning. By representing predictions as density-like matrices and aligning their pairwise relations via MCE, the approach extends traditional cross-entropy with a principled, geometry-aware objective. Empirically, it yields strong gains across CIFAR-10/100 and STL-10, including notable improvements over state-of-the-art methods and compatibility with Curriculum Pseudo Labeling (CPL). The framework rests on solid connections to density matrices, von Neumann entropy, and information geometry, offering a versatile foundation for incorporating relational cues in SSL and beyond.

Abstract

Semi-supervised learning has emerged as a pivotal approach for leveraging scarce labeled data alongside abundant unlabeled data. Despite significant progress, prevailing SSL methods predominantly enforce consistency between different augmented views of individual samples, thereby overlooking the rich relational structure inherent within a mini-batch. In this paper, we present RelationMatch, a novel SSL framework that explicitly enforces in-batch relational consistency through a Matrix Cross-Entropy (MCE) loss function. The proposed MCE loss is rigorously derived from both matrix analysis and information geometry perspectives, ensuring theoretical soundness and practical efficacy. Extensive empirical evaluations on standard benchmarks, including a notable 15.21% accuracy improvement over FlexMatch on STL-10, demonstrate that RelationMatch not only advances state-of-the-art performance but also provides a principled foundation for incorporating relational cues in SSL.

RelationMatch: Matching In-batch Relationships for Semi-supervised Learning

TL;DR

RelationMatch introduces Matrix Cross-Entropy (MCE), a theoretically grounded loss that enforces in-batch relational consistency between weak and strong augmentations in semi-supervised learning. By representing predictions as density-like matrices and aligning their pairwise relations via MCE, the approach extends traditional cross-entropy with a principled, geometry-aware objective. Empirically, it yields strong gains across CIFAR-10/100 and STL-10, including notable improvements over state-of-the-art methods and compatibility with Curriculum Pseudo Labeling (CPL). The framework rests on solid connections to density matrices, von Neumann entropy, and information geometry, offering a versatile foundation for incorporating relational cues in SSL and beyond.

Abstract

Semi-supervised learning has emerged as a pivotal approach for leveraging scarce labeled data alongside abundant unlabeled data. Despite significant progress, prevailing SSL methods predominantly enforce consistency between different augmented views of individual samples, thereby overlooking the rich relational structure inherent within a mini-batch. In this paper, we present RelationMatch, a novel SSL framework that explicitly enforces in-batch relational consistency through a Matrix Cross-Entropy (MCE) loss function. The proposed MCE loss is rigorously derived from both matrix analysis and information geometry perspectives, ensuring theoretical soundness and practical efficacy. Extensive empirical evaluations on standard benchmarks, including a notable 15.21% accuracy improvement over FlexMatch on STL-10, demonstrate that RelationMatch not only advances state-of-the-art performance but also provides a principled foundation for incorporating relational cues in SSL.
Paper Structure (27 sections, 17 theorems, 54 equations, 1 figure, 3 tables)

This paper contains 27 sections, 17 theorems, 54 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Matrix Cross-Entropy for Supervised and Semi-supervised Learning
  3. Warm-up Example
  4. Matrix Cross-Entropy
  5. Matrix Cross-Entropy: Foundations and Interpretations
  6. Density Matrices and the Matrix Logarithm
  7. Von Neumann Entropy and Matrix Cross-Entropy
  8. Information Geometrical Perspective of Matrix Cross-Entropy
  9. Unveiling the Properties of Matrix Cross-Entropy
  10. The Scalar Cross-Entropy as a Special Case of MCE
  11. Desirable Properties of MCE
  12. RelationMatch: Applying MCE for Semi-supervised Learning
  13. Datasets
  14. Experimental Details
  15. Supervised Learning Results
  16. ...and 12 more sections

Key Result

Theorem 3.3

Let $\mathbf{A} \in \mathbb{C}^{n \times n}$ be a matrix with no eigenvalues on $\mathbb{R}^{-}$. Then there exists a unique matrix $\mathbf{X}$ such that with all eigenvalues of $\mathbf{X}$ lying in the strip $\{ z \in \mathbb{C} : -\pi < \operatorname{Im}(z) < \pi \}$. We refer to $\mathbf{X}$ as the principal logarithm of $\mathbf{A}$ and denote it by $\mathbf{X} = \log(\mathbf{A})$. If $\mat

Figures (1)

  • Figure 1: Pseudo-labels are generated by inputting a batch of weakly augmented images into the model. Subsequently, the model computes the probability distributions for the corresponding strongly augmented images. The loss function seamlessly integrates both the traditional cross-entropy loss and the proposed matrix cross-entropy loss.

Theorems & Definitions (23)

  • Definition 2.1: Matrix Cross-Entropy for Positive Semi-Definite Matrices
  • Definition 3.1: Density Matrix on $\mathbb{R}^{n \times n}$
  • Definition 3.2: Matrix Logarithm
  • Theorem 3.3: Principal Matrix Logarithm higham2008functions
  • Proposition 3.4
  • Lemma 3.5
  • Definition 3.6: Matrix Relative Entropy for Density Matrices
  • Theorem 3.7: Projection Theorem amari2014information
  • Proposition 3.8: Minimization Property
  • Lemma 4.1
  • ...and 13 more