Randomized Algorithms for Symmetric Nonnegative Matrix Factorization

TL;DR

This work introduces two randomized approaches for Symmetric Nonnegative Matrix Factorization (SymNMF): LAI-SymNMF, which accelerates computation by first forming a low-rank, symmetric input approximation before solving the factorization, and LvS-SymNMF, which uses leverage-score based sampling to accelerate the sequence of nonnegative least squares subproblems. It provides theoretical guarantees that leverage-score sampling yields provable accuracy for convex least squares problems, including nonnegative least squares, and analyzes a hybrid sampling strategy that deterministically includes high-leverage rows to boost practical performance. Empirically, the methods deliver substantial speedups on large dense and sparse graphs (often 5x–7.5x) while preserving clustering quality, demonstrated on Web of Science and Microsoft Open Academic Graph datasets. The framework unifies randomized linear algebra tools with both alternating-update (ANLS/HALS) and all-at-once (PGNCG) SymNMF solvers, offering a versatile path toward scalable SymNMF for big-data clustering and information fusion tasks. Overall, the paper advances scalable SymNMF by (i) introducing robust randomized input sketching and sampling schemes, (ii) providing rigorous error and complexity analyses, and (iii) validating practical gains on real-world graphs.

Abstract

Symmetric Nonnegative Matrix Factorization (SymNMF) is a technique in data analysis and machine learning that approximates a symmetric matrix with a product of a nonnegative, low-rank matrix and its transpose. To design faster and more scalable algorithms for SymNMF we develop two randomized algorithms for its computation. The first algorithm uses randomized matrix sketching to compute an initial low-rank approximation to the input matrix and proceeds to rapidly compute a SymNMF of the approximation. The second algorithm uses randomized leverage score sampling to approximately solve constrained least squares problems. Many successful methods for SymNMF rely on (approximately) solving sequences of constrained least squares problems. We prove theoretically that leverage score sampling can approximately solve nonnegative least squares problems to a chosen accuracy with high probability. Additionally, we prove sampling complexity results for previously proposed hybrid sampling techniques which deterministically include high leverage score rows. This hybrid scheme is crucial for obtaining speeds ups in practice. Finally we demonstrate that both methods work well in practice by applying them to graph clustering tasks on large real world data sets. These experiments show that our methods approximately maintain solution quality and achieve significant speed ups for both large dense and large sparse problems.

Figures (6)

• Figure 1: Normalized Residual Norm Plots for the WoS data set. Three different update rules are shown: BPP, HALS, and PGNCG.
• Figure 1: Normalized Residual error value for various SymNMF Algorithms on the WoS data set using an EDVW Hypergraph Representation.
• Figure 2: Normalized residual and projected gradient values for OAG using HALS and BPP update rules. $k=16$, $\tau=1/s$ or $\tau=1$ as indicated in the legend.
• Figure 2: Normalized Residual Norm Plots for the WoS data set. Three different update rules are shown: BPP, HALS, and PGNCG. These experiments use $q=2$ and do not make use of \ref{['alg:adaRRF']}.
• Figure 3: Time Breakdown per iteration for three algorithms. Standard SymNMF with HALS update rule (HALS), SymNMF with leverage score sampling and HALS (LvS-HALS), and SymNMF with leverage score sampling and BPP (LvS-BPP).

Theorems & Definitions (16)

• Theorem 2.1
• Proposition 3.1
• Proof 1
• Theorem 3.2
• Proposition 3.3
• Lemma 4.1: qp_pert1
• Proof 2: Proof of Theorem \ref{['thm:nls_bnd']}
• Lemma 4.2
• Proof 3
• Lemma 4.3