Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization

Ruicheng Ao, Hongyu Chen, Haoyang Liu, David Simchi-Levi, Will Wei Sun

TL;DR

This work addresses learning with scarce labels by exploiting predictions from external models as control variates in optimization. It unifies Prediction-Powered Inference (PPI) and Stochastic Variance Reduced Gradient (SVRG) into PPI-SVRG, a variance-reduction framework that uses a snapshot gradient over $N+n$ samples and a prediction-informed control variate. Theoretical results decompose convergence into the standard SVRG-type rate and a prediction-driven error floor that scales with the quality of the predictor; perfect predictions recover SVRG, while imperfect predictions yield a stable neighborhood around the optimum. Empirically, PPI-SVRG achieves substantial mean-estimation improvements under label scarcity (43–52% MSE reduction) and boosts MNIST test accuracy by 2.7–2.9 percentage points with only 10% labeled data, demonstrating practical impact for semi-supervised optimization. The approach is particularly effective when unlabeled data are abundant and predictors are informative, offering a principled path to leveraging pre-trained models in optimization tasks."

Abstract

We study semi-supervised stochastic optimization when labeled data is scarce but predictions from pre-trained models are available. PPI and SVRG both reduce variance through control variates -- PPI uses predictions, SVRG uses reference gradients. We show they are mathematically equivalent and develop PPI-SVRG, which combines both. Our convergence bound decomposes into the standard SVRG rate plus an error floor from prediction uncertainty. The rate depends only on loss geometry; predictions affect only the neighborhood size. When predictions are perfect, we recover SVRG exactly. When predictions degrade, convergence remains stable but reaches a larger neighborhood. Experiments confirm the theory: PPI-SVRG reduces MSE by 43--52\% under label scarcity on mean estimation benchmarks and improves test accuracy by 2.7--2.9 percentage points on MNIST with only 10\% labeled data.

PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization

TL;DR

This work addresses learning with scarce labels by exploiting predictions from external models as control variates in optimization. It unifies Prediction-Powered Inference (PPI) and Stochastic Variance Reduced Gradient (SVRG) into PPI-SVRG, a variance-reduction framework that uses a snapshot gradient over samples and a prediction-informed control variate. Theoretical results decompose convergence into the standard SVRG-type rate and a prediction-driven error floor that scales with the quality of the predictor; perfect predictions recover SVRG, while imperfect predictions yield a stable neighborhood around the optimum. Empirically, PPI-SVRG achieves substantial mean-estimation improvements under label scarcity (43–52% MSE reduction) and boosts MNIST test accuracy by 2.7–2.9 percentage points with only 10% labeled data, demonstrating practical impact for semi-supervised optimization. The approach is particularly effective when unlabeled data are abundant and predictors are informative, offering a principled path to leveraging pre-trained models in optimization tasks."

Abstract

We study semi-supervised stochastic optimization when labeled data is scarce but predictions from pre-trained models are available. PPI and SVRG both reduce variance through control variates -- PPI uses predictions, SVRG uses reference gradients. We show they are mathematically equivalent and develop PPI-SVRG, which combines both. Our convergence bound decomposes into the standard SVRG rate plus an error floor from prediction uncertainty. The rate depends only on loss geometry; predictions affect only the neighborhood size. When predictions are perfect, we recover SVRG exactly. When predictions degrade, convergence remains stable but reaches a larger neighborhood. Experiments confirm the theory: PPI-SVRG reduces MSE by 43--52\% under label scarcity on mean estimation benchmarks and improves test accuracy by 2.7--2.9 percentage points on MNIST with only 10\% labeled data.
Paper Structure (44 sections, 2 theorems, 75 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 44 sections, 2 theorems, 75 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Algorithms
  5. The Connection Between SVRG and PPI
  6. Intuition: Why does the control variate help?
  7. PPI-SVRG++ for General Convex Objectives
  8. Theoretical Analysis
  9. Convergence under Strong Convexity
  10. Convergence under General Convexity
  11. Experiments
  12. Mean Estimation
  13. Semi-Supervised Deep Learning
  14. Conclusion
  15. Proof of \ref{['thm:convergence']} (PPI-SVRG)
  16. ...and 29 more sections

Key Result

Theorem 5.2

Under assump:smooth_convex, for $s\geq 1$, we have where $\alpha = 1/[\gamma\eta(1-2\lambda\eta)m]+2\lambda\eta/(1-2\lambda\eta) < 1$ and $\beta = \eta/(1-2\lambda\eta)$.

Figures (4)

  • Figure 1: Overview of the PPI-SVRG framework. Left: The data landscape---small labeled dataset $(n)$ and large unlabeled dataset $(N)$ with predictions. Center: The mechanism---snapshot gradient (anchor), variance reduction (correction), and epoch doubling (booster). Right: Convergence outcomes---exponential for strongly convex with good predictions, $O(1/T)$ for general convex.
  • Figure 2: Mean estimation on forest: MSE, CI width, and coverage across labeled proportions $\gamma$. PPI-SVRG achieves the lowest MSE and narrowest CIs while maintaining valid coverage. Lower is better for MSE and CI width; 95% is nominal for coverage.
  • Figure 3: Mean estimation on galaxies: PPI-SVRG achieves consistent improvements, with coverage closest to the nominal 95%.
  • Figure 4: Learning curves on MNIST (mean $\pm$ std over 5 seeds). PPI-SVRG variants achieve higher accuracy and lower loss than baselines after an initial warm-up phase.

Theorems & Definitions (7)

  • Theorem 5.2: Convergence of PPI-SVRG
  • Remark 5.3: Extreme Cases
  • Remark 5.4: When is PPI-SVRG Most Useful?
  • Theorem 5.5: Convergence of PPI-SVRG++
  • Remark 5.6: Comparison with Standard SVRG
  • proof
  • proof