Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Worst-Case and Smoothed Analysis of the Hartigan-Wong Method for k-Means Clustering

Bodo Manthey, Jesse van Rhijn

TL;DR

This paper resolves two fundamental questions about the Hartigan–Wong k-means method: (i) it proves an exponential worst-case running time on a simple line instance, even when exact k-means is easy, and (ii) it initiates a rigorous smoothed-analysis bound showing polynomial-time behavior in many regimes under Gaussian perturbations. The exponential lower bound is constructed via a hierarchical sequence of gadgets that force an exponential-length improving path for a carefully chosen pivot rule. The smoothed-analysis result leverages a grid-based center-approximation and a probabilistic sequence bound to show that with high probability, random perturbations eliminate long sequences of tiny improvements, yielding an upper bound of $k^{12kd+5} d^{12} n^{12.5+1/d} \ln^{4.5}(nkd)/\sigma^{4}$. Although not fully polynomial in all parameters, this work advances understanding by matching the spirit of Lloyd’s smoothed results and suggesting that Hartigan–Wong may have polynomial smoothed complexity in broad parameter regimes. The findings bridge theory and practice by explaining why Hartigan–Wong often performs well despite theoretical worst-case exponential behavior and by outlining a path toward tighter smoothed bounds.

Abstract

We analyze the running time of the Hartigan-Wong method, an old algorithm for the $k$-means clustering problem. First, we construct an instance on the line on which the method can take $2^{Ω(n)}$ steps to converge, demonstrating that the Hartigan-Wong method has exponential worst-case running time even when $k$-means is easy to solve. As this is in contrast to the empirical performance of the algorithm, we also analyze the running time in the framework of smoothed analysis. In particular, given an instance of $n$ points in $d$ dimensions, we prove that the expected number of iterations needed for the Hartigan-Wong method to terminate is bounded by $k^{12kd}\cdot poly(n, k, d, 1/σ)$ when the points in the instance are perturbed by independent $d$-dimensional Gaussian random variables of mean $0$ and standard deviation $σ$.

Worst-Case and Smoothed Analysis of the Hartigan-Wong Method for k-Means Clustering

TL;DR

This paper resolves two fundamental questions about the Hartigan–Wong k-means method: (i) it proves an exponential worst-case running time on a simple line instance, even when exact k-means is easy, and (ii) it initiates a rigorous smoothed-analysis bound showing polynomial-time behavior in many regimes under Gaussian perturbations. The exponential lower bound is constructed via a hierarchical sequence of gadgets that force an exponential-length improving path for a carefully chosen pivot rule. The smoothed-analysis result leverages a grid-based center-approximation and a probabilistic sequence bound to show that with high probability, random perturbations eliminate long sequences of tiny improvements, yielding an upper bound of . Although not fully polynomial in all parameters, this work advances understanding by matching the spirit of Lloyd’s smoothed results and suggesting that Hartigan–Wong may have polynomial smoothed complexity in broad parameter regimes. The findings bridge theory and practice by explaining why Hartigan–Wong often performs well despite theoretical worst-case exponential behavior and by outlining a path toward tighter smoothed bounds.

Abstract

We analyze the running time of the Hartigan-Wong method, an old algorithm for the -means clustering problem. First, we construct an instance on the line on which the method can take steps to converge, demonstrating that the Hartigan-Wong method has exponential worst-case running time even when -means is easy to solve. As this is in contrast to the empirical performance of the algorithm, we also analyze the running time in the framework of smoothed analysis. In particular, given an instance of points in dimensions, we prove that the expected number of iterations needed for the Hartigan-Wong method to terminate is bounded by when the points in the instance are perturbed by independent -dimensional Gaussian random variables of mean and standard deviation .
Paper Structure (8 sections, 11 theorems, 18 equations, 2 figures, 1 table)

This paper contains 8 sections, 11 theorems, 18 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries and Notation
  3. Exponential Lower Bound
  4. Formal Construction
  5. Smoothed Analysis
  6. Technical Preliminaries
  7. Approximating Iterations
  8. Analyzing Sequences

Key Result

Theorem 1

For each $m \in \mathbb{N}_{\geq 2}$ there exists an instance of $k$-means clustering on the line with $n = 4m-3$ points and $k = 2m - 1$ clusters on which the Hartigan--Wong method can take $2^{\Omega(n)}$ iterations to converge to a local optimum.

Figures (2)

  • Figure 1: Schematic depiction of the interactions between $G_i$ and $G_{i-1}$ during the morning and afternoon phases of $G_i$.
  • Figure 2: Schematic depiction of the interactions between $G_i$, $G_{i-1}$ and $G_{i+1}$ during the wakeup phase of $G_i$. Note that the final state of $G_i$ corresponds to the first state depicted in \ref{['fig:morning_afternoon']}.

Theorems & Definitions (11)

  • Theorem 1: restate=lowerbound
  • Theorem 2: restate=smoothedcomplexity
  • Lemma 3: Telgarsky & Vattani telgarskyHartiganMethodKmeans2010
  • Lemma 4: Telgarsky & Vattani telgarskyHartiganMethodKmeans2010
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • Lemma 8
  • Lemma 8
  • Lemma 9
  • ...and 1 more