Weighted Low-rank Approximation via Stochastic Gradient Descent on Manifolds
Conglong Xu, Peiqi Yang, Hao Wu
TL;DR
The paper tackles the NP-hard problem of weighted low-rank approximation by reframing it as a manifold optimization problem using a reduced SVD: $P=U D^{k\times k}_k(\mathbf{x}) V^T$ with $U\in V_k(\mathbb{R}^m)$, $V\in V_k(\mathbb{R}^n)$ and $\mathbf{x}\in\mathbb{R}^k$. It develops a stochastic gradient descent on the product manifold with retractions and a confinement mechanism to guarantee convergence, establishing a central convergence theorem (Theorem \ref{['thm-confined-SGD']}) and its corollaries. The framework is instantiated for the reformulated WLRA problem, deriving gradient expressions and a convergent algorithm (Algorithm \ref{['alg-confined-SGD-RWLRA-1-direct']}) that outperforms Euclidean-space SGD on a Netflix-prize data subset, and an accelerated line-search variant on manifolds that accelerates convergence relative to Euclidean baselines. The work also provides extensive theoretical groundwork, including a new proof related to the Eckart-Young theorem, and supplies practical guidance for parameter choices and implementations.
Abstract
We solve a regularized weighted low-rank approximation problem by a stochastic gradient descent on a manifold. To guarantee the convergence of our stochastic gradient descent, we establish a convergence theorem on manifolds for retraction-based stochastic gradient descents admitting confinements. On sample data from the Netflix Prize training dataset, our algorithm outperforms the existing stochastic gradient descent on Euclidean spaces. We also compare the accelerated line search on this manifold to the existing accelerated line search on Euclidean spaces.