Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Supervised Learning Really That Different from Unsupervised?

Oskar Allerbo, Thomas B. Schön

TL;DR

The paper reframes supervised learning as a two-stage process in which model parameters are chosen through unsupervised means and outputs are incorporated afterward, by expressing predictors as linear smoothers: $\hat{f}^* = \mathbf{s}^{*\top} \mathbf{y}$. It introduces a model-selection criterion based on second-sample moment matching (SSMM) that can be made independent of $\mathbf{y}$ (SSMM-Tr) and analyzes its asymptotic behavior in linear ridge regression, showing near-optimal risk in the high-dimensional limit. The authors extend the approach to neural networks via the neural tangent kernel, enabling y-free training by using random labels during optimization, and demonstrate through experiments that y-free methods perform comparably to traditional, label-informed counterparts on several tasks, while substantially outperforming random guessing. The work highlights a close relationship between supervised and unsupervised learning, suggesting that the optimal predictor can be viewed as a weighted sum of $\mathbf{y}$ with weights determined by $\mathbf{X}$ and $\mathbf{x}^*$, and discusses practical caveats related to computation and scaling. Overall, the paper provides a principled, model-agnostic framework for y-free training and offers evidence that the boundary between supervised and unsupervised learning may be more about representation than about fundamental differences in learning signals.

Abstract

We demonstrate how supervised learning can be decomposed into a two-stage procedure, where (1) all model parameters are selected in an unsupervised manner, and (2) the outputs y are added to the model, without changing the parameter values. This is achieved by a new model selection criterion that, in contrast to cross-validation, can be used also without access to y. For linear ridge regression, we bound the asymptotic out-of-sample risk of our method in terms of the optimal asymptotic risk. We also demonstrate on real and synthetic data that versions of linear and kernel ridge regression, smoothing splines, and neural networks, which are trained without access to y, perform similarly to their standard y-based counterparts. Hence, our results suggest that the difference between supervised and unsupervised learning is less fundamental than it may appear.

Is Supervised Learning Really That Different from Unsupervised?

TL;DR

The paper reframes supervised learning as a two-stage process in which model parameters are chosen through unsupervised means and outputs are incorporated afterward, by expressing predictors as linear smoothers: . It introduces a model-selection criterion based on second-sample moment matching (SSMM) that can be made independent of (SSMM-Tr) and analyzes its asymptotic behavior in linear ridge regression, showing near-optimal risk in the high-dimensional limit. The authors extend the approach to neural networks via the neural tangent kernel, enabling y-free training by using random labels during optimization, and demonstrate through experiments that y-free methods perform comparably to traditional, label-informed counterparts on several tasks, while substantially outperforming random guessing. The work highlights a close relationship between supervised and unsupervised learning, suggesting that the optimal predictor can be viewed as a weighted sum of with weights determined by and , and discusses practical caveats related to computation and scaling. Overall, the paper provides a principled, model-agnostic framework for y-free training and offers evidence that the boundary between supervised and unsupervised learning may be more about representation than about fundamental differences in learning signals.

Abstract

We demonstrate how supervised learning can be decomposed into a two-stage procedure, where (1) all model parameters are selected in an unsupervised manner, and (2) the outputs y are added to the model, without changing the parameter values. This is achieved by a new model selection criterion that, in contrast to cross-validation, can be used also without access to y. For linear ridge regression, we bound the asymptotic out-of-sample risk of our method in terms of the optimal asymptotic risk. We also demonstrate on real and synthetic data that versions of linear and kernel ridge regression, smoothing splines, and neural networks, which are trained without access to y, perform similarly to their standard y-based counterparts. Hence, our results suggest that the difference between supervised and unsupervised learning is less fundamental than it may appear.
Paper Structure (21 sections, 8 theorems, 103 equations, 4 figures, 2 tables)

This paper contains 21 sections, 8 theorems, 103 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Selection Without Using the Outputs
  4. Limitations of Model Selection Based on In-Sample Model Complexity
  5. Model Selection Based on Out-of-Sample Predictions: Second Sample Moment Matching
  6. Making Model Selection y-free
  7. When Does y-free Model Selection Work?
  8. What About y-free Cross Validation?
  9. y-free Training of Neural Networks
  10. Experiments
  11. Conclusions
  12. Experimental Details
  13. Details for Figures \ref{['fig:compl_demo']} and \ref{['fig:wo_y_demo']}
  14. Details for the Real Data in Section \ref{['sec:exps']}
  15. y-free Training of Neural Networks in More Detail
  16. ...and 6 more sections

Key Result

Proposition 1

For $\bm{\hat{\beta}}:=\left(\bm{\Phi}^\top\bm{\Phi}+n\lambda\bm{I}_p\right)^{-1}\bm{\Phi}^\top\bm{y}$ and $\bm{s}^*(\lambda):=\bm{\varphi}^{*\top}\left(\bm{\Phi}^\top\bm{\Phi}+n\lambda\bm{I}_p\right)^{-1}\bm{\Phi}^\top$,

Figures (4)

  • Figure 1: Illustrating training without labels: After the label information ($a$, $b$) has been removed, an unsupervised algorithm is used to split the input space into four classes, separated by the purple cross, without using the label information ($c$). After fixing the class borders, the labels are revealed, enabling the model to label each class ($d$).
  • Figure 2: Illustrating the limitations of in-sample model complexity. All three function estimates have the same in-sample model complexity, in terms of the effective number of parameters, but behave very differently between the observations. Details are given in Appendix \ref{['sec:exp_dets']}.
  • Figure 3: Left: Asymptotic risks for different values of $\gamma$ and SNR. For $\overline{R_{\bm{X}}}(\overline{\lambda_T})$ we use solid, for $\overline{R_{\bm{X}}}(\overline{\lambda^*})$ dashed, and for $\overline{R_{\bm{X}}}(0)$ dotted lines. For $\gamma\notin \left(\frac{1}{2},2\right)$, $\overline{R_{\bm{X}}}(\overline{\lambda_T})$ coincides $\overline{R_{\bm{X}}}(0)$, but for $\gamma\in \left(\frac{1}{2},2\right)$, where $\overline{R_{\bm{X}}}(0)$ tends to deviate substantially form $\overline{R_{\bm{X}}}(\overline{\lambda^*})$ (and even diverge for $\gamma=1$), $\overline{R_{\bm{X}}}(\overline{\lambda_T})$ stays much closer to $\overline{R_{\bm{X}}}(\overline{\lambda^*})$. Right: $\overline{R_{\bm{X}}}(\overline{\lambda_T})/\overline{R_{\bm{X}}}(\overline{\lambda^*})$ for SNR in $[1,80]$. In this interval, the quotient is always less than 2.45 (and, for some values of SNR, close to 1), which is expected from Theorem \ref{['thm:mar_pas2']}.
  • Figure 4: The results of training different models with and without $\bm{y}$ on synthetic data. The models trained without $\bm{y}$ generalize as well as those trained with $\bm{y}$. In all six cases, the models are expressed in the form $\hat{f}^*=\bm{s}^{*\top}\bm{y}$, where, in the bottom panel, $\bm{s}^*$ is constructed independently of $\bm{y}$.

Theorems & Definitions (23)

  • Proposition 1
  • Theorem 1
  • Remark 1
  • Theorem 2
  • Remark 2
  • Theorem 3
  • Remark 3
  • Remark 4
  • Remark 5
  • Remark 6
  • ...and 13 more