Faster Algorithms for Structured Linear and Kernel Support Vector Machines
Yuzhou Gu, Zhao Song, Lichen Zhang
TL;DR
The paper develops nearly-linear time algorithms for structured quadratic programs, with a focus on linear and Gaussian kernel SVMs, by introducing a robust interior-point method (IPM) and a suite of data-structures for efficient central-path maintenance. It treats two structural regimes: (i) low-treewidth QPs, using sparse Cholesky factorization and implicit representations to achieve near-linear time in the number of points n; and (ii) low-rank QPs, applying the Woodbury formula to maintain inverses efficiently and achieve time roughly n(k+m)^{(ω+1)/2}. For Gaussian kernel SVMs, the authors combine a rank-n^{o(1)} factorization of the Gaussian kernel with their QP solver to obtain an almost-linear time algorithm under favorable radius B and dimension d, while proving SETH-based hardness when B is large. The results offer both algorithmic advances for SVMs under structure and conditional lower bounds clarifying the difficulty of fast sub-quadratic kernels as data becomes more complex. Overall, the work advances the theory and practice of speeding convex QP/SVM solvers via robust IPM and structure-exploiting data structures.
Abstract
Quadratic programming is a ubiquitous prototype in convex programming. Many machine learning problems can be formulated as quadratic programming, including the famous Support Vector Machines (SVMs). Linear and kernel SVMs have been among the most popular models in machine learning over the past three decades, prior to the deep learning era. Generally, a quadratic program has an input size of $Θ(n^2)$, where $n$ is the number of variables. Assuming the Strong Exponential Time Hypothesis ($\textsf{SETH}$), it is known that no $O(n^{2-o(1)})$ time algorithm exists when the quadratic objective matrix is positive semidefinite (Backurs, Indyk, and Schmidt, NeurIPS'17). However, problems such as SVMs usually admit much smaller input sizes: one is given $n$ data points, each of dimension $d$, and $d$ is oftentimes much smaller than $n$. Furthermore, the SVM program has only $O(1)$ equality linear constraints. This suggests that faster algorithms are feasible, provided the program exhibits certain structures. In this work, we design the first nearly-linear time algorithm for solving quadratic programs whenever the quadratic objective admits a low-rank factorization, and the number of linear constraints is small. Consequently, we obtain results for SVMs: * For linear SVM when the input data is $d$-dimensional, our algorithm runs in time $\widetilde O(nd^{(ω+1)/2}\log(1/ε))$ where $ω\approx 2.37$ is the fast matrix multiplication exponent; * For Gaussian kernel SVM, when the data dimension $d = {\color{black}O(\log n)}$ and the squared dataset radius is sub-logarithmic in $n$, our algorithm runs in time $O(n^{1+o(1)}\log(1/ε))$. We also prove that when the squared dataset radius is at least $Ω(\log^2 n)$, then $Ω(n^{2-o(1)})$ time is required. This improves upon the prior best lower bound in both the dimension $d$ and the squared dataset radius.