Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semi-Supervised Regression with Heteroscedastic Pseudo-Labels

Xueqing Sun, Renzhen Wang, Quanziang Wang, Yichen Wu, Xixi Jia, Deyu Meng

TL;DR

This work tackles semi-supervised regression with continuous outputs by modeling pseudo-label uncertainty as heteroscedastic noise and learning it via a bi-level optimization framework. An inner loop updates the regression model with labeled data and uncertainty-weighted pseudo-labels, while an outer loop trains a lightweight uncertainty-learner to calibrate these weights, aligning inner and outer gradients for better generalization. Theoretical analysis shows gradient alignment under this bi-level setup, and extensive experiments on UTKFace, IMDB-WIKI, and STS-B demonstrate robust improvements, particularly when labeled data are scarce. The method achieves state-of-the-art or competitive performance with modest computational overhead and offers a practical, uncertainty-aware approach to SSR that reduces error propagation from unreliable pseudo-labels.

Abstract

Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete and confidence-based filtering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As a result, naive pseudo-labeling can lead to error accumulation and overfitting to incorrect labels. To address this, we propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty estimates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and extensive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared to existing methods. Our code is available at https://github.com/sxq/Heteroscedastic-Pseudo-Labels.

Semi-Supervised Regression with Heteroscedastic Pseudo-Labels

TL;DR

This work tackles semi-supervised regression with continuous outputs by modeling pseudo-label uncertainty as heteroscedastic noise and learning it via a bi-level optimization framework. An inner loop updates the regression model with labeled data and uncertainty-weighted pseudo-labels, while an outer loop trains a lightweight uncertainty-learner to calibrate these weights, aligning inner and outer gradients for better generalization. Theoretical analysis shows gradient alignment under this bi-level setup, and extensive experiments on UTKFace, IMDB-WIKI, and STS-B demonstrate robust improvements, particularly when labeled data are scarce. The method achieves state-of-the-art or competitive performance with modest computational overhead and offers a practical, uncertainty-aware approach to SSR that reduces error propagation from unreliable pseudo-labels.

Abstract

Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete and confidence-based filtering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As a result, naive pseudo-labeling can lead to error accumulation and overfitting to incorrect labels. To address this, we propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty estimates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and extensive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared to existing methods. Our code is available at https://github.com/sxq/Heteroscedastic-Pseudo-Labels.
Paper Structure (29 sections, 1 theorem, 21 equations, 6 figures, 7 tables, 1 algorithm)

This paper contains 29 sections, 1 theorem, 21 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Semi-Supervised Learning
  4. Semi-Supervised Regression
  5. Uncertainty Estimation
  6. Method
  7. Notations and Problem Setups
  8. Heteroscedastic Pseudo-Labels
  9. Learning Uncertainty via Bi-level Optimization
  10. Practical Implementations
  11. Theoretical Analysis
  12. Experiments
  13. Experimental Setups
  14. Main Results
  15. Discussion and Ablation Study
  16. ...and 14 more sections

Key Result

Theorem 1

Let $\nabla_\theta \mathcal{L}^{inner}(\theta, \phi)$ and $\nabla_{\theta} \mathcal{L}^{outer}(\theta)$ be the gradients of the inner and outer loss w.r.t. $\theta$, respectively. The bi-level optimization in eq:inner_loss and eq:outer_loss is equivalent to the following optimization problem:

Figures (6)

  • Figure 1: Experiment of UCVME dai2023semi on UTKFace zhang2017age with 10% labeled data. Left: Histogram of true labels for the unlabeled data. Right: Box-plot of pseudo-labels generated by UCVME.
  • Figure 2: Method Overview. The proposed bi-level optimization framework consists of two main steps: (1) Inner-loop update, which updates the regression model using $\mathcal{L}^{inner}$ as defined in \ref{['eq:inner_loss']}, where $z_j = \log \sigma_j^2$; (2) Outer-loop update, which updates the uncertainty-learner using $\mathcal{L}^{outer}$ as defined in \ref{['eq:outer_loss']}. Note that we assume a batch size of 1 for better visualization.
  • Figure 3: Subgroup performance comparison between ours and the second-best (UCVME) on UTKFace with $\gamma=5\%$.
  • Figure 4: Correlation analysis between estimated uncertainty $\sigma^2$ and absolute prediction error on IMDB-WIKI with $\gamma=10\%$ under age 40.
  • Figure 5: Correlation analysis between estimated uncertainty $\sigma^2$ and absolute prediction error on IMDB-WIKI with $\gamma = 10\%$ under different age.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof