Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Randomized Nyström Preconditioned Interior Point-Proximal Method of Multipliers

Ya-Chi Chu, Luiz-Rafael Santos, Madeleine Udell

TL;DR

This work tackles the challenge of solving large-scale convex separable quadratic programs with high precision by developing Nys-IP-PMM, a matrix-free, regularized interior-point method that embeds a randomized Nyström preconditioner within the inner CG solver. The method extends IP-PMM to inexact Newton solves, and provides convergence guarantees in a probabilistic setting, including a polynomial-time-like bound on iterations. Empirically, Nyström preconditioning yields faster wallclock performance than partial Cholesky preconditioning, especially for dense constraint matrices, and the authors provide an open-source Julia implementation. The approach broadens the applicability of high-accuracy interior-point methods to large-scale problems in finance and machine learning, offering scalable performance and the potential for hardware acceleration.

Abstract

We present a new algorithm for convex separable quadratic programming (QP) called Nys-IP-PMM, a regularized interior-point solver that uses low-rank structure to accelerate solution of the Newton system. The algorithm combines the interior point proximal method of multipliers (IP-PMM) with the randomized Nyström preconditioned conjugate gradient method as the inner linear system solver. Our algorithm is matrix-free: it accesses the input matrices solely through matrix-vector products, as opposed to methods involving matrix factorization. It works particularly well for separable QP instances with dense constraint matrices. We establish convergence of Nys-IP-PMM. Numerical experiments demonstrate its superior performance in terms of wallclock time compared to previous matrix-free IPM-based approaches.

Randomized Nyström Preconditioned Interior Point-Proximal Method of Multipliers

TL;DR

This work tackles the challenge of solving large-scale convex separable quadratic programs with high precision by developing Nys-IP-PMM, a matrix-free, regularized interior-point method that embeds a randomized Nyström preconditioner within the inner CG solver. The method extends IP-PMM to inexact Newton solves, and provides convergence guarantees in a probabilistic setting, including a polynomial-time-like bound on iterations. Empirically, Nyström preconditioning yields faster wallclock performance than partial Cholesky preconditioning, especially for dense constraint matrices, and the authors provide an open-source Julia implementation. The approach broadens the applicability of high-accuracy interior-point methods to large-scale problems in finance and machine learning, offering scalable performance and the potential for hardware acceleration.

Abstract

We present a new algorithm for convex separable quadratic programming (QP) called Nys-IP-PMM, a regularized interior-point solver that uses low-rank structure to accelerate solution of the Newton system. The algorithm combines the interior point proximal method of multipliers (IP-PMM) with the randomized Nyström preconditioned conjugate gradient method as the inner linear system solver. Our algorithm is matrix-free: it accesses the input matrices solely through matrix-vector products, as opposed to methods involving matrix factorization. It works particularly well for separable QP instances with dense constraint matrices. We establish convergence of Nys-IP-PMM. Numerical experiments demonstrate its superior performance in terms of wallclock time compared to previous matrix-free IPM-based approaches.
Paper Structure (48 sections, 7 theorems, 64 equations, 4 figures, 6 tables, 6 algorithms)

This paper contains 48 sections, 7 theorems, 64 equations, 4 figures, 6 tables, 6 algorithms.

Table of Contents

  1. Key Words.
  2. MSC Codes.
  3. Introduction
  4. Contributions
  5. Comparison to other variants of IP-PMM
  6. Organization
  7. Notation
  8. Background
  9. Interior-point methods (IPMs)
  10. Regularized IPMs
  11. Interior point proximal method of multipliers (IP-PMM)
  12. Preconditioned conjugate gradient
  13. Randomized IPMs for linear programming
  14. Partial Cholesky preconditioners
  15. Randomized Nyström preconditioners
  16. ...and 33 more sections

Key Result

Theorem 3.1

Let $\epsilon \in (0,1)$ be a given error tolerance. Choose a starting point as in eq:starting-pt for inexact IP-PMM and let $C$ and $\omega$ be positive constants such that $\mu_0 \leq \frac{C}{\epsilon^{\omega}}$. Assume that at each iteration, the residuals in eq:inexact-Newton-RHS satisfy assump

Figures (4)

  • Figure 1: Relative primal/dual infeasibility and optimality gap versus cumulative time for portfolio optimization problem with $n = 80000.0$, $d = 50000.0$, and $s = 100$.
  • Figure 2: The condition numbers before and after preconditioning. The subplots represent distinct stages of IP-PMM convergence. The red dashed line shows the unpreconditioned condition number $\kappa(N_{\delta, k})$. Blue circles denote the condition number after partial Cholesky preconditioning, while orange triangles represent the condition number after Nyström preconditioning.
  • Figure 3: Runtime of Nys-IP-PMM and Chol-IP-PMM with varying rank $\ell$ on RNASeq dataset. Left plot compares the two methods; right plot zooms in on the bar chart for Nys-IP-PMM. Bar height show total runtime, which is broken into 1) PCG runtime, 2) construction time for preconditioner, and 3) other computation (which is negligible). Blue number above bar gives percentage PCG time.
  • Figure 4: Distribution of eigenvalues for $AA^T$ and $N_k$ at different IP-PMM iterations of the SVM problem formed from 1000 samples of CIFAR10.

Theorems & Definitions (15)

  • Remark 1: sparsity of $L_{21}$
  • Theorem 3.1
  • proof
  • Remark 2
  • Proposition 3.1
  • Theorem 4.1
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • ...and 5 more