Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A cutting plane algorithm for globally solving low dimensional k-means clustering problems

Martin Ryner, Jan Kronqvist, Johan Karlsson

TL;DR

This work tackles the NP-hard problem of globally solving k-means for low-dimensional data by reformulating it as a concave assignment problem and solving it via a cutting-plane framework that alternates between a small concave subproblem and a large linear program. The authors prove that the upper-lower bound gap converges to zero, ensuring global optimality, and introduce acceleration techniques (symmetry breaking, branching, integer and tight constraints) to make the approach practical. They demonstrate substantial performance gains on synthetic data and show improved alignment with ground truth on MNIST when using the global solution versus local methods. The method provides explicit optimality gaps to terminate early and offers a principled framework for exact clustering in settings with small $k$ and $d$, with future work aimed at polytope refinements and parallelization for larger-scale problems.

Abstract

Clustering is one of the most fundamental tools in data science and machine learning, and k-means clustering is one of the most common such methods. There is a variety of approximate algorithms for the k-means problem, but computing the globally optimal solution is in general NP-hard. In this paper we consider the k-means problem for instances with low dimensional data and formulate it as a structured concave assignment problem. This allows us to exploit the low dimensional structure and solve the problem to global optimality within reasonable time for large data sets with several clusters. The method builds on iteratively solving a small concave problem and a large linear programming problem. This gives a sequence of feasible solutions along with bounds which we show converges to zero optimality gap. The paper combines methods from global optimization theory to accelerate the procedure, and we provide numerical results on their performance.

A cutting plane algorithm for globally solving low dimensional k-means clustering problems

TL;DR

This work tackles the NP-hard problem of globally solving k-means for low-dimensional data by reformulating it as a concave assignment problem and solving it via a cutting-plane framework that alternates between a small concave subproblem and a large linear program. The authors prove that the upper-lower bound gap converges to zero, ensuring global optimality, and introduce acceleration techniques (symmetry breaking, branching, integer and tight constraints) to make the approach practical. They demonstrate substantial performance gains on synthetic data and show improved alignment with ground truth on MNIST when using the global solution versus local methods. The method provides explicit optimality gaps to terminate early and offers a principled framework for exact clustering in settings with small and , with future work aimed at polytope refinements and parallelization for larger-scale problems.

Abstract

Clustering is one of the most fundamental tools in data science and machine learning, and k-means clustering is one of the most common such methods. There is a variety of approximate algorithms for the k-means problem, but computing the globally optimal solution is in general NP-hard. In this paper we consider the k-means problem for instances with low dimensional data and formulate it as a structured concave assignment problem. This allows us to exploit the low dimensional structure and solve the problem to global optimality within reasonable time for large data sets with several clusters. The method builds on iteratively solving a small concave problem and a large linear programming problem. This gives a sequence of feasible solutions along with bounds which we show converges to zero optimality gap. The paper combines methods from global optimization theory to accelerate the procedure, and we provide numerical results on their performance.
Paper Structure (25 sections, 2 theorems, 30 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 25 sections, 2 theorems, 30 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Contributions
  4. Related work
  5. Introduction to the method and structure of the paper
  6. Problem formulation and proposed algorithm
  7. Formulation as a low dimensional concave problem
  8. Relaxation and iterative scheme
  9. Convergence
  10. Accelerating methods
  11. One fixed marginal constraint
  12. Symmetry breaking
  13. Branching
  14. Constrained centroid positioning
  15. Integer constraints
  16. ...and 10 more sections

Key Result

Proposition 2.1

The k-means problem eq:problemformulation2 is equivalent to the concave problem in the sense that there is a $\Gamma\in \mathcal{A}$ that is minimizes both problems, and their corresponding minimum values coincide.

Figures (3)

  • Figure 1: A convex set (ellipse drawn with a solid line) is approximated by with linear cutting planes tangential to the convex set. A concave objective function (with level curves described by thin dashed ellipses) attains its minimum in an extreme point $E$ of the outer linear approximation which is lower than any value in the ellipse. A new constraint (thick dashed line), refines the outer approximation by removing a part which contains $E$.
  • Figure 2: A model problem with simulated data. Left: Exact solution to the k-means problem using the proposed algorithm. Right: Approximate solution of the problem using k-means++.
  • Figure 3: Clustering differences using exact solution of the k-means problem and k-means++. The images show the two most significant dimensions of the multi dimensional scaling. Left image: Ground truth, middle image: Exact solution, right image: k-means++. The exact solution tends to follow the ground truth, whilst the k-means++ has found a local optimum which is not describing the clusters correctly.

Theorems & Definitions (4)

  • Proposition 2.1
  • Remark 2.2
  • Theorem 3.1
  • proof