Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization

Zhong Ji, Shuo Yang, Jingren Liu, Yanwei Pang, Jungong Han

TL;DR

A Non-Negative Generalized Category Discovery framework that employs Symmetric Non-negative Matrix Factorization as a mathematical medium to prove the equivalence of optimal K-means with optimal SNMF, and reframes the optimization of $\bar{A}$ and K-means clustering as an NCL optimization problem.

Abstract

Generalized Category Discovery (GCD) aims to classify both base and novel images using labeled base data. However, current approaches inadequately address the intrinsic optimization of the co-occurrence matrix $\bar{A}$ based on cosine similarity, failing to achieve zero base-novel regions and adequate sparsity in base and novel domains. To address these deficiencies, we propose a Non-Negative Generalized Category Discovery (NN-GCD) framework. It employs Symmetric Non-negative Matrix Factorization (SNMF) as a mathematical medium to prove the equivalence of optimal K-means with optimal SNMF, and the equivalence of SNMF solver with non-negative contrastive learning (NCL) optimization. Utilizing these theoretical equivalences, it reframes the optimization of $\bar{A}$ and K-means clustering as an NCL optimization problem. Moreover, to satisfy the non-negative constraints and make a GCD model converge to a near-optimal region, we propose a GELU activation function and an NMF NCE loss. To transition $\bar{A}$ from a suboptimal state to the desired $\bar{A}^*$, we introduce a hybrid sparse regularization approach to impose sparsity constraints. Experimental results show NN-GCD outperforms state-of-the-art methods on GCD benchmarks, achieving an average accuracy of 66.1\% on the Semantic Shift Benchmark, surpassing prior counterparts by 4.7\%.

A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization

TL;DR

A Non-Negative Generalized Category Discovery framework that employs Symmetric Non-negative Matrix Factorization as a mathematical medium to prove the equivalence of optimal K-means with optimal SNMF, and reframes the optimization of and K-means clustering as an NCL optimization problem.

Abstract

Generalized Category Discovery (GCD) aims to classify both base and novel images using labeled base data. However, current approaches inadequately address the intrinsic optimization of the co-occurrence matrix based on cosine similarity, failing to achieve zero base-novel regions and adequate sparsity in base and novel domains. To address these deficiencies, we propose a Non-Negative Generalized Category Discovery (NN-GCD) framework. It employs Symmetric Non-negative Matrix Factorization (SNMF) as a mathematical medium to prove the equivalence of optimal K-means with optimal SNMF, and the equivalence of SNMF solver with non-negative contrastive learning (NCL) optimization. Utilizing these theoretical equivalences, it reframes the optimization of and K-means clustering as an NCL optimization problem. Moreover, to satisfy the non-negative constraints and make a GCD model converge to a near-optimal region, we propose a GELU activation function and an NMF NCE loss. To transition from a suboptimal state to the desired , we introduce a hybrid sparse regularization approach to impose sparsity constraints. Experimental results show NN-GCD outperforms state-of-the-art methods on GCD benchmarks, achieving an average accuracy of 66.1\% on the Semantic Shift Benchmark, surpassing prior counterparts by 4.7\%.

Paper Structure

This paper contains 16 sections, 2 theorems, 32 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Theoretical Insights
  5. Equivalence of Kernel K-Means and SNMF
  6. Equivalence of SNMF and NCL
  7. Method
  8. Non-negative Activated Neurons
  9. NMF NCE Loss
  10. Hybrid Sparse Regularization
  11. Experiments
  12. Comparison with the State-of-the-arts
  13. Ablation Study
  14. Hyper-parameters Study
  15. Visualization
  16. ...and 1 more sections

Key Result

Theorem 1

Let $\{x_i\}_{i=1}^n$ be a dataset transformed into a Reproducing Kernel Hilbert Space (RKHS) via an optimal neural network $f^*$, resulting in optimal kernel matrix $A^* = f^*(X)^\top f^*(X)$, which is unnormalized $\bar{A^*}$. The optimal Kernel K-means objective can be reformulated as: where $H = (h_1, \ldots, h_K)$ with $h_k^\top h_l = \delta_{kl}$, $H^T H = I$, and $A^{*\top} A^* = \text{con

Figures (8)

  • Figure 1: The comparison of a suboptimal co-occurrence matrix $\bar{A}$ and its optimally refined counterpart $\bar{A}^*$, alongside the theoretical underpinnings involved in attaining the optimal $\bar{A}^*$ in GCD scenarios, including their equivalence. The asterisk $*$ symbol denotes that the components have been optimized to their respective optimal conditions.
  • Figure 2: Non-zero activations formed with GELU.
  • Figure 3: An analytical visualization contrasting the features extracted via our proposed methodology with those obtained through SimGCD wen2023parametric.
  • Figure 4: A comparative visualization of the $\bar{A}$ matrix computed from features extracted by our NN-GCD versus those derived using SimGCD wen2023parametric.
  • Figure 5: Results with varying hyperparameters of NMF NCE loss on CUB. From left to right: temperature, $\mu$ and $\sigma$.
  • ...and 3 more figures

Theorems & Definitions (4)

  • Theorem 1: Equivalence of optimal Kernel K-Means and optimal SNMF
  • proof
  • Theorem 2
  • proof