Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA

TL;DR

The paper develops an exact, high-dimensional framework to analyze the transient dynamics of online learning algorithms by showing that the time-evolving joint empirical measures converge to a deterministic measure-valued process governed by nonlinear PDEs.Applying this framework to online regularized regression and online PCA reveals precise PDE characterizations of the dynamics, including a decoupled 1-D effective coordinate behavior under exchangeability, and provides tractable numerical methods for PDE solutions to predict algorithm performance.Central contributions include a general meta-theorem for exchangeable Markov chains, weak-convergence results to measure-valued PDEs, and a practical interpretation of dynamics as 1-D stochastic gradient actions in effective energy landscapes, with implications for nonconvex optimization and adaptive learning.

Abstract

We present a framework for analyzing the exact dynamics of a class of online learning algorithms in the high-dimensional scaling limit. Our results are applied to two concrete examples: online regularized linear regression and principal component analysis. As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measures of the target feature vector and its estimates provided by the algorithms will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE. Numerical solutions of this PDE can be efficiently obtained. These solutions lead to precise predictions of the performance of the algorithms, as many practical performance metrics are linear functionals of the joint empirical measures. In addition to characterizing the dynamic performance of online learning algorithms, our asymptotic analysis also provides useful insights. In particular, in the high-dimensional limit, and due to exchangeability, the original coupled dynamics associated with the algorithms will be asymptotically "decoupled", with each coordinate independently solving a 1-D effective minimization problem via stochastic gradient descent. Exploiting this insight for nonconvex optimization problems may prove an interesting line of future research.

Figures (5)

• Figure 1: Time-varying probability densities of stochastic gradient descent for minimizing a 1-D nonconvex function. In the scaling limit, the densities $p_t(x)$ are the solution of a deterministic Fokker-Planck equation given in \ref{['eq:pde_1d']}.
• Figure 2: Asymptotic predictions v.s. simulations results. The red solid curves are predictions of the probability density $\pi_t(x \, \vert \, \xi=1)$ given by the PDE \ref{['eq:pde_lasso_strong']}, and the blue bars show the empirical histograms of the estimates obtained by the online regularized regression algorithm at four different times. The signal dimension in this experiment is $n = 10^5$.
• Figure 3: The mean square error (MSE) v.s. $t=k/n$: We run $100$ independent trials of the online learning algorithm for the regularized linear regression problem. The error bars show confidence intervals of one standard deviation. The result indicates that the empirical MSE curves converge to a deterministic one as $n$ increases. Moreover, this limit curve is accurately predicted by our asymptotic characterization.
• Figure 4: The trade-off between the true positive and false positive rates for sparse support estimation. The limiting measure as specified by the PDE \ref{['eq:pde_weak_pca']} can accurately predict the exact trade-off at any given time $t$.
• Figure 5: Dynamics of the regularized regression algorithm using a convex regularizer $\Phi(x)=|x|$ (the first row) and a nonconvex regularizer $\Phi(x)=\tanh(\alpha\left|x\right|)$ (the second row). The first column shows the two regularizers. The second and third columns show the effective 1-D potential \ref{['eq:opt-1d']} for $\xi=0$ and $\xi = 3$, respectively. The fourth column shows the MSE v.s. regularization strength $\beta$ at different iteration times $t$. The learning rate $\tau=0.2$ is fixed. The figures show a nonconvex regularizer may have a better performance. However, inappropriate algorithmic parameters can make the dynamics trapped in a metastable state for a very long time when another local minimum emerges in the 1-D potential function.

Theorems & Definitions (37)

• Example 1: Regularized linear regression
• Example 2: Regularized PCA
• Theorem 1
• Remark 1
• Proposition 1
• Example 3
• Theorem 2
• Remark 2
• Example 4: Support Recovery
• Remark 3