Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orthogonal Nonnegative Matrix Factorization with the Kullback-Leibler divergence

Jean Pacifique Nkurunziza, Fulgence Nahayo, Nicolas Gillis

TL;DR

The paper addresses clustering with ONMF under a KL divergence objective, suitable for nonnegative, Poisson-like data such as word-count vectors and certain imaging data. It develops KL-ONMF using alternating optimization with closed-form updates: W is updated as a cluster-wise average of data columns, and H is updated by selecting active clusters per data point via a KL-consistent criterion, ensuring HH^⊤ = I_r. Empirically, KL-ONMF achieves higher clustering accuracy on document datasets and improved material endmember extraction in hyperspectral unmixing compared to Frobenius-norm ONMF, with competitive run times and favorable convergence. The approach is scalable, with a per-iteration cost on the order of O(nnz(X) r), and the authors provide code and discuss extending the framework to additional divergences for broader applicability.

Abstract

Orthogonal nonnegative matrix factorization (ONMF) has become a standard approach for clustering. As far as we know, most works on ONMF rely on the Frobenius norm to assess the quality of the approximation. This paper presents a new model and algorithm for ONMF that minimizes the Kullback-Leibler (KL) divergence. As opposed to the Frobenius norm which assumes Gaussian noise, the KL divergence is the maximum likelihood estimator for Poisson-distributed data, which can model better sparse vectors of word counts in document data sets and photo counting processes in imaging. We develop an algorithm based on alternating optimization, KL-ONMF, and show that it performs favorably with the Frobenius-norm based ONMF for document classification and hyperspectral image unmixing.

Orthogonal Nonnegative Matrix Factorization with the Kullback-Leibler divergence

TL;DR

The paper addresses clustering with ONMF under a KL divergence objective, suitable for nonnegative, Poisson-like data such as word-count vectors and certain imaging data. It develops KL-ONMF using alternating optimization with closed-form updates: W is updated as a cluster-wise average of data columns, and H is updated by selecting active clusters per data point via a KL-consistent criterion, ensuring HH^⊤ = I_r. Empirically, KL-ONMF achieves higher clustering accuracy on document datasets and improved material endmember extraction in hyperspectral unmixing compared to Frobenius-norm ONMF, with competitive run times and favorable convergence. The approach is scalable, with a per-iteration cost on the order of O(nnz(X) r), and the authors provide code and discuss extending the framework to additional divergences for broader applicability.

Abstract

Orthogonal nonnegative matrix factorization (ONMF) has become a standard approach for clustering. As far as we know, most works on ONMF rely on the Frobenius norm to assess the quality of the approximation. This paper presents a new model and algorithm for ONMF that minimizes the Kullback-Leibler (KL) divergence. As opposed to the Frobenius norm which assumes Gaussian noise, the KL divergence is the maximum likelihood estimator for Poisson-distributed data, which can model better sparse vectors of word counts in document data sets and photo counting processes in imaging. We develop an algorithm based on alternating optimization, KL-ONMF, and show that it performs favorably with the Frobenius-norm based ONMF for document classification and hyperspectral image unmixing.

Paper Structure

This paper contains 19 sections, 2 theorems, 17 equations, 3 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Outline and contribution of the paper
  3. Notation
  4. Alternating optimization for ONMF with the Frobenius norm
  5. Computational cost
  6. Convergence
  7. Stopping criterion
  8. Alternating optimization for ONMF with the KL divergence
  9. Update $W$ when $H$ is fixed
  10. Update $H$ when $W$ is fixed
  11. Algorithm KL-ONMF
  12. Computational cost
  13. Convergence
  14. Stopping criterion
  15. Numerical Experiments
  16. ...and 4 more sections

Key Result

Lemma 1

The optimal solution of the problem eq24_2_1 is given by

Figures (3)

  • Figure 1: Spectral signatures extracted with KL-ONMF and Fro-ONMF on hyperspectral images (these corresponds to the columns of the computed $W$ factor); from top to bottom: Moffet, Samson and Jasper.
  • Figure 2: Clustering of the Moffet HSI (from left to right: water, soil and vegetation), and the Samson HSI (from left to right: soil, tree, water).
  • Figure 3: Clustering of the Jasper HSI. From left to right: tree, water, soil and road.

Theorems & Definitions (4)

  • Lemma 1
  • proof
  • Lemma 2
  • proof