Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast, Scalable, Warm-Start Semidefinite Programming with Spectral Bundling and Sketching

Rico Angell, Andrew McCallum

TL;DR

USBS introduces Unified Spectral Bundling with Sketching, a fast, scalable SDP solver that unifies spectral bundle methods for general SDPs with both equality and inequality constraints and augments them with Nyström-based matrix sketching. The core idea is to minimize a penalized dual objective in a low-dimensional spectral subspace, updating the subspace and the candidate dual iterate without requiring iteration-dependent step sizes, and to recover a high-quality primal solution through a controlled low-rank reconstruction. The authors prove non-asymptotic convergence rates (sublinear in general, improving to near-linear under favorable regularity) and demonstrate remarkable empirical performance on MaxCut, Quadratic Assignment, and interactive entity resolution problems, including warm-start scenarios that yield large speedups. The practical impact is substantial: the approach enables solving SDPs with billions to trillions of variables, reduces memory footprints, and provides a concrete, hardware-friendly implementation in pure JAX for CPU/GPU/TPU, broadening the applicability of SDPs in large-scale, real-time, and data-incremental settings.

Abstract

While semidefinite programming (SDP) has traditionally been limited to moderate-sized problems, recent algorithms augmented with matrix sketching techniques have enabled solving larger SDPs. However, these methods achieve scalability at the cost of an increase in the number of necessary iterations, resulting in slower convergence as the problem size grows. Furthermore, they require iteration-dependent parameter schedules that prohibit effective utilization of warm-start initializations important in practical applications with incrementally-arriving data or mixed-integer programming. We present Unified Spectral Bundling with Sketching (USBS), a provably correct, fast and scalable algorithm for solving massive SDPs that can leverage a warm-start initialization to further accelerate convergence. Our proposed algorithm is a spectral bundle method for solving general SDPs containing both equality and inequality constraints. Moveover, when augmented with an optional matrix sketching technique, our algorithm achieves the dramatically improved scalability of previous work while sustaining convergence speed. We empirically demonstrate the effectiveness of our method across multiple applications, with and without warm-starting. For example, USBS provides a 500x speed-up over the state-of-the-art scalable SDP solver on an instance with over 2 billion decision variables.

Fast, Scalable, Warm-Start Semidefinite Programming with Spectral Bundling and Sketching

TL;DR

USBS introduces Unified Spectral Bundling with Sketching, a fast, scalable SDP solver that unifies spectral bundle methods for general SDPs with both equality and inequality constraints and augments them with Nyström-based matrix sketching. The core idea is to minimize a penalized dual objective in a low-dimensional spectral subspace, updating the subspace and the candidate dual iterate without requiring iteration-dependent step sizes, and to recover a high-quality primal solution through a controlled low-rank reconstruction. The authors prove non-asymptotic convergence rates (sublinear in general, improving to near-linear under favorable regularity) and demonstrate remarkable empirical performance on MaxCut, Quadratic Assignment, and interactive entity resolution problems, including warm-start scenarios that yield large speedups. The practical impact is substantial: the approach enables solving SDPs with billions to trillions of variables, reduces memory footprints, and provides a concrete, hardware-friendly implementation in pure JAX for CPU/GPU/TPU, broadening the applicability of SDPs in large-scale, real-time, and data-incremental settings.

Abstract

While semidefinite programming (SDP) has traditionally been limited to moderate-sized problems, recent algorithms augmented with matrix sketching techniques have enabled solving larger SDPs. However, these methods achieve scalability at the cost of an increase in the number of necessary iterations, resulting in slower convergence as the problem size grows. Furthermore, they require iteration-dependent parameter schedules that prohibit effective utilization of warm-start initializations important in practical applications with incrementally-arriving data or mixed-integer programming. We present Unified Spectral Bundling with Sketching (USBS), a provably correct, fast and scalable algorithm for solving massive SDPs that can leverage a warm-start initialization to further accelerate convergence. Our proposed algorithm is a spectral bundle method for solving general SDPs containing both equality and inequality constraints. Moveover, when augmented with an optional matrix sketching technique, our algorithm achieves the dramatically improved scalability of previous work while sustaining convergence speed. We empirically demonstrate the effectiveness of our method across multiple applications, with and without warm-starting. For example, USBS provides a 500x speed-up over the state-of-the-art scalable SDP solver on an instance with over 2 billion decision variables.
Paper Structure (55 sections, 6 theorems, 105 equations, 14 figures, 3 tables, 1 algorithm)

This paper contains 55 sections, 6 theorems, 105 equations, 14 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Preliminaries
  4. Semidefinite Programming
  5. Proximal Bundle Method
  6. Model Requirements.
  7. Unified Spectral Bundling
  8. Spectral Bundle Model
  9. Defining the model.
  10. Updating the model.
  11. Computing the Candidate Iterate
  12. Feasibility of $\tilde{y}_{t+1}$.
  13. Convergence Rate.
  14. Scaling with Matrix Sketching
  15. Experiments
  16. ...and 40 more sections

Key Result

Theorem 3.1

Suppose strong duality holds. For any fixed $\rho > 0, \beta \in (0, 1), k_c \geq 1, k_p \geq 0$, and $\alpha \geq 2 \mathfrak{N}(\mathcal{X}_\star)$, USBS produces iterates $X_t \succeq 0$ and $y_t \in \mathcal{Y}$ such that for any $\varepsilon \in (0, 1]$, by some iteration $\mathcal{O}(1 / \varepsilon^3)$. And, if strict complementarity holds, then these conditions are achieved by some iterat

Figures (14)

  • Figure 1: Convergence time (sec) to moderate relative error tolerance ($\downarrow$). The time (in seconds) for CGAL and USBS to achieve an $\varepsilon$-approximate solution for $\varepsilon = 10^{-1}$ with and without warm-starting on 99% of initial data on ten DIMACS10 MaxCut instances. The bars marked with $\dagger$ indicate an $\varepsilon$-approximate solution was not achieved in 72 hours. The datasets are sorted in ascending order by $n$, ranging from 16K to 3.7M (more than $10^{13}$ decision variables for 333SP). Note that warm-starting generally improves convergence, but does not always (e.g. 598a). We observe USBS achieves an extraordinary improvement in convergence over CGAL which fails to reach an accurate solution on 7 out of 10 instances. In contrast, USBS is able to reach a solution on all of the problem instances in 28 hours or less without a warm-start initialization.
  • Figure 2: relative gap ($\downarrow$) vs. time. We plot the relative gap (y-axis, left) and best relative gap (y-axis, right) against time in seconds (x-axis) for one QAP instance, pr144, from TSPLIB ($\texttt{\small n} = 144$) over one hour of optimization. We observe that for both algorithms the best rounded solution is found early in optimization. We observe that USBS is able to leverage a warm-start initialization.
  • Figure 3: Cumulative SDP solve time ($\downarrow$) vs. number of $\exists$-constraints . In each plot (one for each author coreference dataset), the $x$-axis is the number of $\exists$-constraints generated one after the other over time and the $y$-axis is the the cumulative solve time (in seconds) for each SDP solver to reach a relative suboptimality, relative infeasibility, and max absolute infeasibility (i.e. $\|\mathcal{A} X - \mathrm{proj}_\mathcal{K}(\mathcal{A} X)\|_\infty \leq \varepsilon$) of $\varepsilon = 10^{-1}$. When warm-starting, both solvers are initialized using the solution from the previous SDP (with one less $\exists$-constraint ). $\exists$-constraints are generated until the perfect clustering is predicted. We observe that USBS is able to leverage a warm-start initialization. In addition, we observe that the performance gap between USBS and CGAL grows as the problem size grows. See \ref{['tab:ecc_dataset_stats']} for dataset sizes and details.
  • Figure 4: Convergence measures on instance144. We solve instance 144 from DIMACS10 and plot the primal objective sub-optimality (objective residual, $\downarrow$), relative infeasibility (infeasibility gap, $\downarrow$), and weight of the cut ($\uparrow$) produced by the rounding procedure. The warm-started runs use 99% of the original data to obtain a warm-start initialization. We observe that USBS is able to more reliably leverage a warm-start initialization. In these plots, USBS is executed with $k_c = 8, k_p = 8$. All runs were executed on a compute node with 16 cores and 128GB of RAM.
  • Figure 5: Infeasibility gap vs. time for different settings of $k_c$ and $k_p$. In each plot, the x-axis is time (up to 12 hours) and the y-axis is the relative infeasibility gap. In every case, USBS is cold-started on the 144 instance from the DIMACS10 dataset. Each plot considers one value of $k_p$ and several values of $k_c$. All runs were executed on a compute node with 16 cores and 128GB of RAM. We observe that USBS performs best when $k_c \geq k_p$.
  • ...and 9 more figures

Theorems & Definitions (11)

  • Theorem 3.1
  • Theorem D.1
  • proof
  • Lemma D.3
  • proof
  • Lemma D.4
  • proof
  • Lemma D.5
  • proof
  • Lemma D.6
  • ...and 1 more