Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation

TL;DR

This paper tackles weighted low rank approximation, a problem known to be NP-hard, by developing an efficient alternating minimization framework that allows approximate updates. Central to the approach is a high-accuracy regression solver based on sketching (SRHT) that preconditions the regression problems and enables nearly linear per-iteration time, with the analysis demonstrating robustness to forward-approximate updates. The authors prove a main result: under standard assumptions on incoherence and spectral gaps, the algorithm converges in $O(\log(1/\epsilon))$ iterations and achieves a spectral-norm error of $O(\alpha^{-1} k\tau)\|W\circ N\| + \epsilon$, with time $\widetilde{O}((\|W\|_0\,k + nk^3)\log(1/\epsilon))$ per iteration. This bridges theory and practice by accommodating inexact solvers while preserving convergence guarantees, and it yields substantial speedups over previous exact-regression-based methods. The work has practical impact on problems like weighted matrix completion and related low-rank recovery tasks where sparsity in the weight and scalability are crucial.

Abstract

Weighted low rank approximation is a fundamental problem in numerical linear algebra, and it has many applications in machine learning. Given a matrix $M \in \mathbb{R}^{n \times n}$, a non-negative weight matrix $W \in \mathbb{R}_{\geq 0}^{n \times n}$, a parameter $k$, the goal is to output two matrices $X,Y\in \mathbb{R}^{n \times k}$ such that $\| W \circ (M - X Y^\top) \|_F$ is minimized, where $\circ$ denotes the Hadamard product. It naturally generalizes the well-studied low rank matrix completion problem. Such a problem is known to be NP-hard and even hard to approximate assuming the Exponential Time Hypothesis [GG11, RSW16]. Meanwhile, alternating minimization is a good heuristic solution for weighted low rank approximation. In particular, [LLR16] shows that, under mild assumptions, alternating minimization does provide provable guarantees. In this work, we develop an efficient and robust framework for alternating minimization that allows the alternating updates to be computed approximately. For weighted low rank approximation, this improves the runtime of [LLR16] from $\|W\|_0k^2$ to $\|W\|_0 k$ where $\|W\|_0$ denotes the number of nonzero entries of the weight matrix. At the heart of our framework is a high-accuracy multiple response regression solver together with a robust analysis of alternating minimization.

Theorems & Definitions (78)

• Theorem 1.1: Informal version of Theorem \ref{['thm:main']}
• Remark 1.2
• Definition 2.1: SRHT ldfu13
• Definition 2.2: Oblivious subspace embedding s06
• Lemma 4.1
• Lemma 4.2
• Lemma 4.3
• Definition 4.4
• Lemma 4.5
• Theorem 4.6: Formal version of Theorem \ref{['thm:main_informal']}