Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Role of Pseudo-labels in Self-training Linear Classifiers on High-dimensional Gaussian Mixture Data

Takashi Takahashi

TL;DR

This work develops a sharp, high-dimensional analysis of iterative self-training with pseudo-labels for binary Gaussian mixtures by applying the replica method and replica-symmetric saddle-point theory. It reveals two regimes: a few iterations where ST benefits come from aligning with reliable pseudo-labels and larger parameter updates, and many iterations where ST gradually aligns the classifier direction through small updates and soft labels, effectively extracting information with minimal noise. The analysis explains why label imbalance degrades ST relative to supervised learning and proposes two heuristics—pseudo-label annealing and bias-fixing—that restore performance to near-supervised levels even with significant imbalance. Numerical experiments validate the RS predictions, and the continuum-limit perturbative analysis provides insights into dynamics and stability, suggesting practical guidelines for hyperparameter tuning and when to apply ST with PLS or annealing. The framework offers a principled route to understand and improve ST in high-dimensional, convex-loss settings and points to extensions to richer models and alternative SSL strategies.

Abstract

Self-training (ST) is a simple yet effective semi-supervised learning method. However, why and how ST improves generalization performance by using potentially erroneous pseudo-labels is still not well understood. To deepen the understanding of ST, we derive and analyze a sharp characterization of the behavior of iterative ST when training a linear classifier by minimizing the ridge-regularized convex loss on binary Gaussian mixtures, in the asymptotic limit where input dimension and data size diverge proportionally. The results show that ST improves generalization in different ways depending on the number of iterations. When the number of iterations is small, ST improves generalization performance by fitting the model to relatively reliable pseudo-labels and updating the model parameters by a large amount at each iteration. This suggests that ST works intuitively. On the other hand, with many iterations, ST can gradually improve the direction of the classification plane by updating the model parameters incrementally, using soft labels and small regularization. It is argued that this is because the small update of ST can extract information from the data in an almost noiseless way. However, in the presence of label imbalance, the generalization performance of ST underperforms supervised learning with true labels. To overcome this, two heuristics are proposed to enable ST to achieve nearly compatible performance with supervised learning even with significant label imbalance.

The Role of Pseudo-labels in Self-training Linear Classifiers on High-dimensional Gaussian Mixture Data

TL;DR

This work develops a sharp, high-dimensional analysis of iterative self-training with pseudo-labels for binary Gaussian mixtures by applying the replica method and replica-symmetric saddle-point theory. It reveals two regimes: a few iterations where ST benefits come from aligning with reliable pseudo-labels and larger parameter updates, and many iterations where ST gradually aligns the classifier direction through small updates and soft labels, effectively extracting information with minimal noise. The analysis explains why label imbalance degrades ST relative to supervised learning and proposes two heuristics—pseudo-label annealing and bias-fixing—that restore performance to near-supervised levels even with significant imbalance. Numerical experiments validate the RS predictions, and the continuum-limit perturbative analysis provides insights into dynamics and stability, suggesting practical guidelines for hyperparameter tuning and when to apply ST with PLS or annealing. The framework offers a principled route to understand and improve ST in high-dimensional, convex-loss settings and points to extensions to richer models and alternative SSL strategies.

Abstract

Self-training (ST) is a simple yet effective semi-supervised learning method. However, why and how ST improves generalization performance by using potentially erroneous pseudo-labels is still not well understood. To deepen the understanding of ST, we derive and analyze a sharp characterization of the behavior of iterative ST when training a linear classifier by minimizing the ridge-regularized convex loss on binary Gaussian mixtures, in the asymptotic limit where input dimension and data size diverge proportionally. The results show that ST improves generalization in different ways depending on the number of iterations. When the number of iterations is small, ST improves generalization performance by fitting the model to relatively reliable pseudo-labels and updating the model parameters by a large amount at each iteration. This suggests that ST works intuitively. On the other hand, with many iterations, ST can gradually improve the direction of the classification plane by updating the model parameters incrementally, using soft labels and small regularization. It is argued that this is because the small update of ST can extract information from the data in an almost noiseless way. However, in the presence of label imbalance, the generalization performance of ST underperforms supervised learning with true labels. To overcome this, two heuristics are proposed to enable ST to achieve nearly compatible performance with supervised learning even with significant label imbalance.
Paper Structure (48 sections, 124 equations, 13 figures, 2 tables, 1 algorithm)

This paper contains 48 sections, 124 equations, 13 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Notations
  4. Problem setup
  5. Generalization error
  6. Sharp asymptotics of ST
  7. statistical mechanics formulation of ST
  8. Chain of Boltzmann distributions
  9. Generating functional
  10. Replica method for ST
  11. Handling of the replicated system
  12. RS solution
  13. RS saddle point
  14. Generating functional for the model parameter
  15. Generalization error
  16. ...and 33 more sections

Figures (13)

  • Figure 1: (a)-(f): Comparison between the empirical distribution of the elements of $\hat{\bm{w}}^{(t)}$ (histogram), which is obtained by single-shot numerical experiment of finite size, and the theoretical prediction given by the Gaussian process $\hat{\mathsf{w}}^{(t)}$ in \ref{['eq: GP of w']} (solid line). (g)-(l): Comparison between the empirical distribution of the elements of $\{\hat{u}_\nu^{(t)}=\hat{\bm{w}}^{(t)}\cdot \bm{x}_\nu^{(t)}/\sqrt{N}+\hat{B}^{(t)}\}_{\nu=1}^{M_U}$ (histogram), which is obtained by single-shot numerical experiment of finite size, and the theoretical prediction given by $\hat{\mathsf{u}}^{(t)} + h_u^{(t)}$ in \ref{['eq:rs-saddle-uhat']} and \ref{['eq: intuitive hu2']} (solid line). Different colors represent different iteration steps of ST with total number of iterations $T=16$. For the details of the settings, refer to the main text.
  • Figure 2: Comparison of the macroscopic quantities obtained by experiments of finite-size systems (markers with error bars) and the theoretical prediction (black solid line). The error bars represent standard deviations. (a)-(c): The squared norm of the weight vector: $\|\hat{\bm{w}}^{(t)}\|_2^2/N$ and $q^{(t)}$ in \ref{['eq: meaning of q']}. (d)-(f): The inner product between the cluster center and the weight vector: $\bm{v}\cdot \hat{\bm{w}}^{(t)}/N$ and $m^{(t)}$ in \ref{['eq: meaning of m']}. (g)-(i): The bias: $\hat{B}^{(t)}$ and $B^{(t)}$ in \ref{['eq: meaning of B']}. (j)-(l): Generalization error: $\epsilon_{\rm g}^{(t)}$ defined in \ref{['eq: gen_err']} and that in \ref{['eq: rs generalization error']}.
  • Figure 3: The ratio of the generalization error obtained at the end of ST \ref{['eq: rs generalization error']} ($t=T$) to the SL with a labeled dataset of size $N(\alpha_L + \alpha_U \times T)$. When the ratio is unity, the ST with an unlabeled dataset has the same performance as the SL with labeled data of the same size. Different colors and lines represent different pairs of $(\rho, \Delta)$. The filled markers with solid lines represent the result with PLS, where $\Gamma$ is optimized so that $\epsilon_{\rm g}^{(T)}$ is minimized. The white markers with dashed lines represent the result without PLS ($\Gamma=0$). The upper panel shows the raw values. The lower panel shows their absolute values of the deviation from unity in the log scale.
  • Figure 4: The the cosine similarity \ref{['eq: cosine similarity']} obtained at the end of ST \ref{['eq: rs generalization error']} ($t=T$). When the value is unity, the classification plane is oriented in the optimal direction. Different colors and lines represent different pairs of $(\rho, \alpha_L, \alpha_U)$. The filled markers with solid lines represent the result with PLS, where $\Gamma$ is optimized so that $\epsilon_{\rm g}^{(T)}$ is minimized. The white markers with dashed lines represent the result without PLS ($\Gamma=0$). The upper panel shows the raw values. The lower panel shows their absolute values of the deviation from unity in the log scale.
  • Figure 5: Comparison of generalization error with and without the implementation of PLS. Upper panel shows the result including label imbalanced cases. Lower panel shows the result of label balanced cases only.
  • ...and 8 more figures

Theorems & Definitions (9)

  • Definition 1: Self-consistent equations
  • Claim 1
  • Claim 2
  • Claim 3
  • Claim 4
  • Claim 5
  • Claim 6
  • Claim 7
  • Claim 8