Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolators

TL;DR

This paper develops a theory and methodology for mixed semi-supervised regression, showing that incorporating unlabeled data with a positive mixing ratio $\alpha$ improves predictive performance for generalized linear models and linear interpolators. It introduces two mixed estimators, derives bias-variance decompositions, and proves that optimal mixing ratios lie strictly between 0 and 1, with closed-form or easily estimable expressions. The approach is extended to over-parameterized interpolators and integrated into deep learning contexts by treating the last layer (or a shallow surrogate) as a linear predictor trained with mixed losses, with extensive simulations and real-data experiments (CelebA, Netflix) demonstrating substantial predictive gains. Practical data-driven procedures to estimate $\alpha$ are provided, including estimators for noise and signal, and asymptotic analyses guarantee the benefits persist as dimensions grow. Overall, the mixed-SSL framework offers a principled, broadly applicable means to leverage unlabeled data for regression, including deep learning settings, with tangible improvements in real-world tasks.

Abstract

We present a methodology for using unlabeled data to design semi-supervised learning (SSL) methods that improve the predictive performance of supervised learning for regression tasks. The main idea is to design different mechanisms for integrating the unlabeled data, and include in each of them a mixing parameter $α$, controlling the weight given to the unlabeled data. Focusing on Generalized Linear Models (GLM) and linear interpolators classes of models, we analyze the characteristics of different mixing mechanisms, and prove that it is consistently beneficial to integrate the unlabeled data with some nonzero mixing ratio $α>0$, in terms of predictive performance. Moreover, we provide a rigorous framework to estimate the best mixing ratio where mixed-SSL delivers the best predictive performance, while using the labeled and unlabeled data on hand. The effectiveness of our methodology in delivering substantial improvement compared to the standard supervised models, in a variety of settings, is demonstrated empirically through extensive simulation, providing empirical support for our theoretical analysis. We also demonstrate the applicability of our methodology (with some heuristic modifications) to improve more complex models, such as deep neural networks, in real-world regression tasks

Figures (10)

• Figure 1: Mean predictive errors of supervised and semi-supervised estimators in ELU model with $n=50$ and $p=10$. On the left side, we examine various values of $\sigma^2$. On the right side, $\sigma^2=25$ and we examine various values of $\alpha$ along the $[0,1]$ grid. see text for details.
• Figure 2: Mean reducible predictive errors of LM with constant-$\beta$ mechanism, for various values of $\sigma^2$, with $n=100$ and $p=50$ (see text for details).
• Figure 3: Relative reducible predictive errors of the random-$\beta$ mechanism for various values of $n$, with $\gamma=0.5$, and $\text{tr}(\Sigma)=\sigma^2=25$ (see text for details).
• Figure 4: Mean predictive errors of ELU model, for various values of $\sigma^2$, with $n=50$ and $p=10$ (see text for details).
• Figure 5: Mean predictive errors of ELU model, for various values of $\alpha$, with $\sigma^2=25$, $n=50$ and $p=10$ (see text for details).

Theorems & Definitions (10)

• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Proposition 3.1
• Theorem 3.1
• Proposition 4.1
• Proposition 5.1
• Theorem 5.1
• Proposition 5.2
• Theorem 5.2