Prediction-Powered Semi-Supervised Learning with Online Power Tuning
Noa Shoham, Ron Dorfman, Shalev Shaer, Kfir Y. Levy, Yaniv Romano
TL;DR
PP-SSL tackles pseudo-label bias in semi-supervised learning by deriving an unbiased gradient estimator that combines labeled and pseudo-labeled data via $g_{PP}^{\lambda}$. It extends Prediction-Powered Inference to SSL and introduces online tuning of the interpolation weight $\lambda$ using AdaGrad, yielding finite-time convergence guarantees that approach the offline optimum $\lambda^* = \tfrac{1}{1+r}\cdot\tfrac{\sigma^2}{\sigma^2+\sigma_e^2}$. Theoretical results bound the gradient variance in terms of teacher error $\mathcal{E}^f$ and data ratio $r$, and show convergence rates of $\mathcal{O}(\sqrt{V^*/T})$ with online adaptation. Empirically, PP-SSL with online tuning outperforms standard SSL and PPI-based methods on synthetic and real datasets, especially when the teacher underperforms on a subgroup, and achieves faster convergence. This work offers a principled, scalable framework for robust SSL under imperfect pseudo-labels with practical online parameter tuning.
Abstract
Prediction-Powered Inference (PPI) is a recently proposed statistical inference technique for parameter estimation that leverages pseudo-labels on both labeled and unlabeled data to construct an unbiased, low-variance estimator. In this work, we extend its core idea to semi-supervised learning (SSL) for model training, introducing a novel unbiased gradient estimator. This extension addresses a key challenge in SSL: while unlabeled data can improve model performance, its benefit heavily depends on the quality of pseudo-labels. Inaccurate pseudo-labels can introduce bias, leading to suboptimal models.To balance the contributions of labeled and pseudo-labeled data, we utilize an interpolation parameter and tune it on the fly, alongside the model parameters, using a one-dimensional online learning algorithm. We verify the practical advantage of our approach through experiments on both synthetic and real datasets, demonstrating improved performance over classic SSL baselines and PPI methods that tune the interpolation parameter offline.