Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conic-Optimization Based Algorithms for Nonnegative Matrix Factorization

Valentin Leplat, Yurii Nesterov, Nicolas Gillis, François Glineur

TL;DR

The paper tackles the NP-hard problem of nonnegative matrix factorization by introducing two conic optimization formulations: an exponential-cone-based under-approximation and a rotated-second-order-cone-based over-approximation. It then develops a successive linearization (majorization-minimization) algorithm that solves convex conic subproblems at each iteration and proves a $\mathcal{O}(1/i)$ convergence rate in the Frank–Wolfe gap to stationary points. The authors show that, in the rank-one case, one formulation becomes convex and solvable efficiently, and they demonstrate through numerical experiments that the methods frequently yield exact NMFs and compare favorably with state-of-the-art approaches on several matrix classes, albeit with scalability limitations due to interior-point methods. The work offers a principled, convergent framework for exact NMF and points to future improvements in initialization, scalability, and extensions to related factorization problems.

Abstract

Nonnegative matrix factorization is the following problem: given a nonnegative input matrix $V$ and a factorization rank $K$, compute two nonnegative matrices, $W$ with $K$ columns and $H$ with $K$ rows, such that $WH$ approximates $V$ as well as possible. In this paper, we propose two new approaches for computing high-quality NMF solutions using conic optimization. These approaches rely on the same two steps. First, we reformulate NMF as minimizing a concave function over a product of convex cones--one approach is based on the exponential cone, and the other on the second-order cone. Then, we solve these reformulations iteratively: at each step, we minimize exactly, over the feasible set, a majorization of the objective functions obtained via linearization at the current iterate. Hence these subproblems are convex conic programs and can be solved efficiently using dedicated algorithms. We prove that our approaches reach a stationary point with an accuracy decreasing as $\mathcal{O}(\frac{1}{i})$, where $i$ denotes the iteration number. To the best of our knowledge, our analysis is the first to provide a convergence rate to stationary points for NMF. Furthermore, in the particular cases of rank-one factorizations (that is, $K=1$), we show that one of our formulations can be expressed as a convex optimization problem implying that optimal rank-one approximations can be computed efficiently. Finally, we show on several numerical examples that our approaches are able to frequently compute exact NMFs (that is, with $V = WH$), and compete favorably with the state of the art.

Conic-Optimization Based Algorithms for Nonnegative Matrix Factorization

TL;DR

The paper tackles the NP-hard problem of nonnegative matrix factorization by introducing two conic optimization formulations: an exponential-cone-based under-approximation and a rotated-second-order-cone-based over-approximation. It then develops a successive linearization (majorization-minimization) algorithm that solves convex conic subproblems at each iteration and proves a convergence rate in the Frank–Wolfe gap to stationary points. The authors show that, in the rank-one case, one formulation becomes convex and solvable efficiently, and they demonstrate through numerical experiments that the methods frequently yield exact NMFs and compare favorably with state-of-the-art approaches on several matrix classes, albeit with scalability limitations due to interior-point methods. The work offers a principled, convergent framework for exact NMF and points to future improvements in initialization, scalability, and extensions to related factorization problems.

Abstract

Nonnegative matrix factorization is the following problem: given a nonnegative input matrix and a factorization rank , compute two nonnegative matrices, with columns and with rows, such that approximates as well as possible. In this paper, we propose two new approaches for computing high-quality NMF solutions using conic optimization. These approaches rely on the same two steps. First, we reformulate NMF as minimizing a concave function over a product of convex cones--one approach is based on the exponential cone, and the other on the second-order cone. Then, we solve these reformulations iteratively: at each step, we minimize exactly, over the feasible set, a majorization of the objective functions obtained via linearization at the current iterate. Hence these subproblems are convex conic programs and can be solved efficiently using dedicated algorithms. We prove that our approaches reach a stationary point with an accuracy decreasing as , where denotes the iteration number. To the best of our knowledge, our analysis is the first to provide a convergence rate to stationary points for NMF. Furthermore, in the particular cases of rank-one factorizations (that is, ), we show that one of our formulations can be expressed as a convex optimization problem implying that optimal rank-one approximations can be computed efficiently. Finally, we show on several numerical examples that our approaches are able to frequently compute exact NMFs (that is, with ), and compete favorably with the state of the art.

Paper Structure

This paper contains 15 sections, 2 theorems, 23 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Computational complexity
  3. Contribution and outline of the paper
  4. NMF formulations based on conic optimization
  5. NMF formulation via exponential cones
  6. NMF formulation via rotated second-order cones
  7. Rank-one Nonnegative Matrix Over-approximation
  8. A successive linearization algorithm
  9. Convergence results
  10. Compactness assumption and convergence of Algorithm \ref{['SCCAE-NMF']}
  11. Numerical experiments
  12. Conclusion
  13. Factorized matrices
  14. Sparsity Patterns Integration
  15. Additional empirical validations of the convergence rates

Key Result

Theorem 4.1

Consider the problem opt_prob where $\Phi$ is a continuously differentiable concave function over the compact convex domain $Q$. Let us denote $Z^{(i)}$ the sequence of iterates generated by the FW algorithm (Algorithm FW) applied on opt_prob. Assume there exists a constant $\tilde{\tau} > 0$ such t where $\Phi(Z^{(0)})-\Phi^*$ is the (global) accuracy of the initial iterate.

Figures (4)

  • Figure 1: Evolution of the minimum FW gap computed along iterations by Algorithm \ref{['SCCAE-NMF']} for \ref{['optiprob2']} (left) and \ref{['optiprob4']} (right) for the matrix ${V_{\text{inf}2}}$.
  • Figure B1: Evolution of $\frac{\left\|V-WH \right\|_F}{\left\| V \right\|_F}$ along iterations; SPI is triggered at iteration $400$.
  • Figure C1: Evolution of the minimum FW gap computed along iterations by Algorithm \ref{['SCCAE-NMF']} for \ref{['optiprob2']} (left) and \ref{['optiprob4']} (right) for a randomly geenrated matrix.
  • Figure C2: Evolution of the minimum FW gap computed along iterations by Algorithm \ref{['SCCAE-NMF']} for \ref{['optiprob2']} (left) and \ref{['optiprob4']} (right) for the matrix ${V_{a=4}}$.

Theorems & Definitions (5)

  • Remark 1: Focus on exact NMF
  • Theorem 4.1
  • Corollary 4.2
  • Remark 2: Difference-of-convex-functions optimization
  • Remark 3