Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lee and Seung (2000)'s Algorithms for Non-negative Matrix Factorization: A Supplementary Proof Guide

Sungjae Cho

TL;DR

This supplementary guide clarifies the formulation and element-wise proof details underlying Lee & Seung's non-negative matrix factorization with multiplicative updates. By deriving MM-based auxiliary functions for both the Euclidean and generalized KL costs, it shows how the multiplicative update rules for W and H monotonically decrease the respective objectives, with invariance at stationary points. The document also connects these updates to gradient-descent viewpoints and provides convergence proofs for both cost functions through majorization-minimization arguments. It further discusses alternative KL formulations and the design rationale for auxiliary functions, highlighting the practical and theoretical significance of multiplicative updates in NNMF. Overall, it deepens understanding of the original proofs and solidifies the theoretical foundation for MU-NMF algorithms used in dimensionality reduction and neural-network learning contexts.

Abstract

Lee and Seung (2000) introduced numerical solutions for non-negative matrix factorization (NMF) using iterative multiplicative update algorithms. These algorithms have been actively utilized as dimensionality reduction tools for high-dimensional non-negative data and learning algorithms for artificial neural networks. Despite a considerable amount of literature on the applications of the NMF algorithms, detailed explanations about their formulation and derivation are lacking. This report provides supplementary details to help understand the formulation and derivation of the proofs as used in the original paper.

Lee and Seung (2000)'s Algorithms for Non-negative Matrix Factorization: A Supplementary Proof Guide

TL;DR

This supplementary guide clarifies the formulation and element-wise proof details underlying Lee & Seung's non-negative matrix factorization with multiplicative updates. By deriving MM-based auxiliary functions for both the Euclidean and generalized KL costs, it shows how the multiplicative update rules for W and H monotonically decrease the respective objectives, with invariance at stationary points. The document also connects these updates to gradient-descent viewpoints and provides convergence proofs for both cost functions through majorization-minimization arguments. It further discusses alternative KL formulations and the design rationale for auxiliary functions, highlighting the practical and theoretical significance of multiplicative updates in NNMF. Overall, it deepens understanding of the original proofs and solidifies the theoretical foundation for MU-NMF algorithms used in dimensionality reduction and neural-network learning contexts.

Abstract

Lee and Seung (2000) introduced numerical solutions for non-negative matrix factorization (NMF) using iterative multiplicative update algorithms. These algorithms have been actively utilized as dimensionality reduction tools for high-dimensional non-negative data and learning algorithms for artificial neural networks. Despite a considerable amount of literature on the applications of the NMF algorithms, detailed explanations about their formulation and derivation are lacking. This report provides supplementary details to help understand the formulation and derivation of the proofs as used in the original paper.
Paper Structure (48 sections, 77 equations, 3 figures)

This paper contains 48 sections, 77 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Non-negative matrix factorization
  3. NMF problem
  4. Definition
  5. Matrices in practice
  6. Zero elements of $W$ and $H$
  7. Cost functions
  8. Euclidean distance
  9. Kullback-Leibler divergence
  10. Two alternative formulations of NMF as optimization problems
  11. Problem 1
  12. Problem 2
  13. Multiplicative update rules
  14. Theorem 1
  15. Theorem 2
  16. ...and 33 more sections

Figures (3)

  • Figure 1: The landscapes of KL divergence with respect to $h_a$; KLD: KL divergence commonly used; GKLD: generalized KL divergence; left: larger scale; right: smaller scale.
  • Figure 2: The intersection of $f(w_a,h_a) = (v_a-w_a h_a)^2$ where $v_a = 1$ with multiple planes. (a,b) These examples show that $f$ is convex if either $w_a$ or $h_a$ is fixed. (c) This example shows that $f$ can be convex without fixing $h_a$ or $w_a$. (d) This intersection shows a counterexample that proves that $f$ is not always convex. The dots have been used to prove the non-convexity of $f$.
  • Figure 3: The intersection of $f(w_a, h_a) = v_a \log \frac{v_a}{w_a h_a} - v_a + w_a h_a$ where $v_a = 1$ with multiple planes. (a,b) These examples show that $f$ is convex if either $w_a$ or $h_a$ is fixed. (c) This example shows that $f$ can be convex without fixing $h_a$ or $w_a$. (d) This intersection shows a counterexample that proves that $f$ is not always convex. The dots have been used to prove the non-convexity of $f$.