Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral Clustering via Orthogonalization-Free Methods

Qiyuan Pang, Haizhao Yang

TL;DR

Numerical results on the streaming graphs show that the orthogonalization-free methods are competitive in the streaming graph scenario since they can take full advantage of the computed features of previous graphs and converge fast, and are more scalable in parallel computing environments because orthogonalization is unnecessary.

Abstract

While orthogonalization exists in current dimensionality reduction methods in spectral clustering on undirected graphs, it does not scale in parallel computing environments. We propose four orthogonalization-free methods for spectral clustering. Our methods optimize one of two objective functions with no spurious local minima. In theory, two methods converge to features isomorphic to the eigenvectors corresponding to the smallest eigenvalues of the symmetric normalized Laplacian. The other two converge to features isomorphic to weighted eigenvectors weighting by the square roots of eigenvalues. We provide numerical evidence on the synthetic graphs from the IEEE HPEC Graph Challenge to demonstrate the effectiveness of the orthogonalization-free methods. Numerical results on the streaming graphs show that the orthogonalization-free methods are competitive in the streaming graph scenario since they can take full advantage of the computed features of previous graphs and converge fast. Our methods are also more scalable in parallel computing environments because orthogonalization is unnecessary. Numerical results are provided to demonstrate the scalability of our methods. Consequently, our methods have advantages over other dimensionality reduction methods when handling spectral clustering for large streaming graphs.

Spectral Clustering via Orthogonalization-Free Methods

TL;DR

Numerical results on the streaming graphs show that the orthogonalization-free methods are competitive in the streaming graph scenario since they can take full advantage of the computed features of previous graphs and converge fast, and are more scalable in parallel computing environments because orthogonalization is unnecessary.

Abstract

While orthogonalization exists in current dimensionality reduction methods in spectral clustering on undirected graphs, it does not scale in parallel computing environments. We propose four orthogonalization-free methods for spectral clustering. Our methods optimize one of two objective functions with no spurious local minima. In theory, two methods converge to features isomorphic to the eigenvectors corresponding to the smallest eigenvalues of the symmetric normalized Laplacian. The other two converge to features isomorphic to weighted eigenvectors weighting by the square roots of eigenvalues. We provide numerical evidence on the synthetic graphs from the IEEE HPEC Graph Challenge to demonstrate the effectiveness of the orthogonalization-free methods. Numerical results on the streaming graphs show that the orthogonalization-free methods are competitive in the streaming graph scenario since they can take full advantage of the computed features of previous graphs and converge fast. Our methods are also more scalable in parallel computing environments because orthogonalization is unnecessary. Numerical results are provided to demonstrate the scalability of our methods. Consequently, our methods have advantages over other dimensionality reduction methods when handling spectral clustering for large streaming graphs.
Paper Structure (17 sections, 4 theorems, 22 equations, 13 figures, 2 tables, 2 algorithms)

This paper contains 17 sections, 4 theorems, 22 equations, 13 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Orthogonalization-Free Methods
  3. OFM-$f_1$ and TriOFM-$f_1$
  4. OFM-$f_2$ and TriOFM-$f_2$
  5. Implementation Details
  6. Momentum Acceleration
  7. Stepsizes
  8. Parallelization
  9. Complexity Analysis
  10. Numerical Results
  11. Comparisons of features $\mathbf{U}_k$ and $\mathbf{U}_k \sqrt{-\Lambda_{k}}$
  12. Comparisons of OFMs with and without orthogonalization
  13. Comparisons on Static Graphs
  14. Comparisons on Streaming Graphs
  15. Scalability
  16. ...and 2 more sections

Key Result

Theorem 1

All stationary points of $f_{1}$ are of form $\mathbf{X} = \mathbf{U}_{\ell}\sqrt{-\Lambda_{\ell}}\mathbf{S}\mathbf{P}$ and all local minima are of form $\mathbf{X} = \mathbf{U}_{k}\sqrt{-\Lambda_{k}}\mathbf{Q}$, $\mathbf{S} \in \mathbf{R}^{\ell\times \ell}$ is a diagonal matrix with diagonal entrie

Figures (13)

  • Figure 1: Clustering performance (ARI $\&$ NMI) based on eigenvectors $\mathbf{U}_{k}$ and weighted eigenvectors $\mathbf{U}_{k} \sqrt{-\Lambda_k}$ where $\Lambda_k$ contains the smallest eigenvalues of $\mathbf{A}$, on clustering static graphs each with 200 thousand nodes (left) or 1 million nodes (right). Eigenvectors $\mathbf{U}_{k}$ are first computed by LOBPCG with $0.1$ tolerance ('LOBPCG(.1)'), and weighted eigenvectors $\mathbf{U}_{k} \sqrt{-\Lambda_k}$ are then evaluated ('LOBPCG(.1,w)').
  • Figure 2: Comparison between using the direct results of OFM-$f_1$ or TriOFM-$f_1$ and using the eigenvectors evaluated by OFM-$f_1$ or TriOFM-$f_1$ followed by orthogonalization and the Rayleigh-Ritz method, as features in spectral clustering quality. The static graphs used for comparisons have 200 thousand nodes (left) or 1 million nodes (right). 'OFM-$f_1$(30)' indicates the features are the direct results of OFM-$f_1$ running with $30$ iterations. 'OFM-$f_1$(30,o)' indicates the features are eigenvectors computed by 'OFM-$f_1$(30)', orthogonalization, and the Rayleigh-Ritz method. Other notations in the plots have similar meanings.
  • Figure 3: Comparison between using the direct results of OFM-$f_2$ or TriOFM-$f_2$ and using the eigenvectors evaluated by OFM-$f_2$ or TriOFM-$f_2$ followed by orthogonalization and the Rayleigh-Ritz method, as features in spectral clustering quality. The static graphs used for comparisons have 200 thousand nodes (left) or 1 million nodes (right). 'OFM-$f_2$(30)' indicates the features are the direct results of OFM-$f_2$ running with $30$ iterations. 'OFM-$f_2$(30,o)' indicates the features are eigenvectors computed by 'OFM-$f_2$(30)', orthogonalization, and the Rayleigh-Ritz method. Other notations in the plots have similar meanings.
  • Figure 4: Comparison of the orthogonalization-free methods, GSF, and PI, on clustering static graphs each with $1$ million graph nodes.
  • Figure 5: Comparison of the orthogonalization-free methods, ARPACK, LOBPCG with AMG preconditioning, and LOBPCG without preconditioning, on clustering static graphs each with $1$ million graph nodes.
  • ...and 8 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4