Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fundamental Limits of Spectral Clustering in Stochastic Block Models

Anderson Ye Zhang

TL;DR

This work establishes the fundamental limits of spectral clustering for community detection in Stochastic Block Models under extreme sparsity, showing that the clustering error decays exponentially with an exact exponent J_min. The authors develop a novel truncated ℓ2 perturbation analysis of eigenvectors, paired with an eigenvector truncation mechanism and leave-one-out decoupling, to obtain matching upper and lower bounds with identical leading constants (up to small log factors). The main results quantify the precise exponential rate in terms of the model parameters, including the community sizes {n_a}, within-same-community probability p, and cross-community probability q, via J_min. These findings sharply characterize when spectral clustering achieves exact recovery and illuminate the technique's limitations in very sparse regimes, providing a rigorous, instance-wise understanding beyond minimax perspectives.

Abstract

Spectral clustering has been widely used for community detection in network sciences. While its empirical successes are well-documented, a clear theoretical understanding, particularly for sparse networks where degrees are much smaller than $\log n$, remains unclear. In this paper, we address this significant gap by demonstrating that spectral clustering offers exponentially small error rates when applied to sparse networks under Stochastic Block Models. Our analysis provides sharp characterizations of its performance, backed by matching upper and lower bounds possessing an identical exponent with the same leading constant. The key to our results is a novel truncated $\ell_2$ perturbation analysis for eigenvectors, coupled with a new analysis idea of eigenvectors truncation.

Fundamental Limits of Spectral Clustering in Stochastic Block Models

TL;DR

This work establishes the fundamental limits of spectral clustering for community detection in Stochastic Block Models under extreme sparsity, showing that the clustering error decays exponentially with an exact exponent J_min. The authors develop a novel truncated ℓ2 perturbation analysis of eigenvectors, paired with an eigenvector truncation mechanism and leave-one-out decoupling, to obtain matching upper and lower bounds with identical leading constants (up to small log factors). The main results quantify the precise exponential rate in terms of the model parameters, including the community sizes {n_a}, within-same-community probability p, and cross-community probability q, via J_min. These findings sharply characterize when spectral clustering achieves exact recovery and illuminate the technique's limitations in very sparse regimes, providing a rigorous, instance-wise understanding beyond minimax perspectives.

Abstract

Spectral clustering has been widely used for community detection in network sciences. While its empirical successes are well-documented, a clear theoretical understanding, particularly for sparse networks where degrees are much smaller than , remains unclear. In this paper, we address this significant gap by demonstrating that spectral clustering offers exponentially small error rates when applied to sparse networks under Stochastic Block Models. Our analysis provides sharp characterizations of its performance, backed by matching upper and lower bounds possessing an identical exponent with the same leading constant. The key to our results is a novel truncated perturbation analysis for eigenvectors, coupled with a new analysis idea of eigenvectors truncation.
Paper Structure (18 sections, 22 theorems, 228 equations, 1 algorithm)

This paper contains 18 sections, 22 theorems, 228 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Organization.
  3. Notation.
  4. Preliminaries
  5. Community Detection and Stochastic Block Models
  6. Spectral Clustering
  7. A Polynomial Upper Bound
  8. Oracle Analysis and Exponents
  9. Main Results
  10. Upper Bound
  11. Truncated $\ell_2$ Perturbation Analysis for Eigenspaces
  12. Lower Bound
  13. Proofs of Theorem \ref{['thm:upper']} and Theorem \ref{['thm:truncated_l2']}
  14. Proof of Theorem \ref{['thm:upper']}
  15. Proof of Theorem \ref{['thm:truncated_l2']}
  16. ...and 3 more sections

Key Result

Theorem 1

Assume $k$ is a constant and all community sizes $n_1,n_2,\ldots,n_k$ are of the same order. Assume $p,q$ satisfy $0<q<p \leq 1/10$ and are of the same order. We further assume $\frac{n(p-q)^2}{p}\rightarrow\infty$. Then where

Theorems & Definitions (38)

  • Theorem 1
  • Lemma 1
  • Lemma 2
  • Proposition 1
  • Proposition 2
  • Theorem 2
  • Theorem 3
  • Corollary 1
  • Theorem 4
  • Lemma 3
  • ...and 28 more