Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quasi-Self-Concordant Optimization with Lewis Weights

Alina Ene, Ta Duy Nguyen, Adrian Vladu

TL;DR

This work addresses constrained quasi-self-concordant optimization for $h(x)=\sum_{i=1}^{n} f((Ax-b)_i)$ under affine constraints $Nx=v$ in the overdetermined regime $(n\ge d)$. It introduces a trust-region framework where each outer iteration reduces to solving a residual problem that is efficiently handled by a new $\ell_{\infty}$ regression solver built from an IRLS scheme using $\ell_{\infty}$ Lewis weight overestimates. The main theoretical result is that the method achieves $\tilde{O}(d^{1/3})$ outer iterations with $O(d^{1/3}\log n)$ linear-system solves per iteration, improving prior dependence on $n$ and delivering a practical algorithm with strong empirical performance against CVX. The approach extends to underdetermined settings and can handle a broad class of functions whose Hessian is stable within $\ell_{\infty}$ boxes, highlighting both theoretical and practical advances in scalable QSC optimization. Overall, the paper provides a simpler, implementable alternative to prior complex solvers while delivering fast convergence and robust performance in large-scale regression-like problems.

Abstract

In this paper, we study the problem $\min_{x\in \mathbb{R}^{d},Nx=v}\sum_{i=1}^{n}f((Ax-b)_{i})$ for a quasi-self-concordant function $f:\mathbb{R}\to\mathbb{R}$, where $A,N$ are $n\times d$ and $m\times d$ matrices, $b,v$ are vectors of length $n$ and $m$ with $n\ge d.$ We show an algorithm based on a trust-region method with an oracle that can be implemented using $\widetilde{O}(d^{1/3})$ linear system solves, improving the $\widetilde{O}(n^{1/3})$ oracle by {[}Adil-Bullins-Sachdeva, NeurIPS 2021{]}. Our implementation of the oracle relies on solving the overdetermined $\ell_{\infty}$-regression problem $\min_{x\in\mathbb{R}^{d},Nx=v}\|Ax-b\|_{\infty}$. We provide an algorithm that finds a $(1+ε)$-approximate solution to this problem using $O((d^{1/3}/ε+1/ε^{2})\log(n/ε))$ linear system solves. This algorithm leverages $\ell_{\infty}$ Lewis weight overestimates and achieves this iteration complexity via a simple lightweight IRLS approach, inspired by the work of {[}Ene-Vladu, ICML 2019{]}. Experimentally, we demonstrate that our algorithm significantly improves the runtime of the standard CVX solver.

Quasi-Self-Concordant Optimization with Lewis Weights

TL;DR

This work addresses constrained quasi-self-concordant optimization for under affine constraints in the overdetermined regime . It introduces a trust-region framework where each outer iteration reduces to solving a residual problem that is efficiently handled by a new regression solver built from an IRLS scheme using Lewis weight overestimates. The main theoretical result is that the method achieves outer iterations with linear-system solves per iteration, improving prior dependence on and delivering a practical algorithm with strong empirical performance against CVX. The approach extends to underdetermined settings and can handle a broad class of functions whose Hessian is stable within boxes, highlighting both theoretical and practical advances in scalable QSC optimization. Overall, the paper provides a simpler, implementable alternative to prior complex solvers while delivering fast convergence and robust performance in large-scale regression-like problems.

Abstract

In this paper, we study the problem for a quasi-self-concordant function , where are and matrices, are vectors of length and with We show an algorithm based on a trust-region method with an oracle that can be implemented using linear system solves, improving the oracle by {[}Adil-Bullins-Sachdeva, NeurIPS 2021{]}. Our implementation of the oracle relies on solving the overdetermined -regression problem . We provide an algorithm that finds a -approximate solution to this problem using linear system solves. This algorithm leverages Lewis weight overestimates and achieves this iteration complexity via a simple lightweight IRLS approach, inspired by the work of {[}Ene-Vladu, ICML 2019{]}. Experimentally, we demonstrate that our algorithm significantly improves the runtime of the standard CVX solver.
Paper Structure (21 sections, 27 theorems, 75 equations, 1 figure, 1 table, 6 algorithms)

This paper contains 21 sections, 27 theorems, 75 equations, 1 figure, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Related work
  4. Overview of our approach
  5. Preliminaries
  6. $\ell_{\infty}$-Regression with Lewis Weights
  7. Algorithm
  8. Analysis
  9. Quasi-Self-Concordant Optimization
  10. Experiments
  11. Energy Lemmas
  12. Analysis of the $\ell_{\infty}$-Regression Algorithm
  13. Correctness
  14. Runtime
  15. Analysis of the QSC Algorithm
  16. ...and 6 more sections

Key Result

Theorem 1.1

Let $f:\mathbb{\mathbb{R}}\to\mathbb{\mathbb{R}}$ be a $C$-quasi-self-concordant function, let $A\in\mathbb{\mathbb{R}}^{n\times d}$, $N\in\mathbb{\mathbb{R}}^{m\times d}$, $b\in\mathbb{\mathbb{R}}^{n}$ and $v\in\mathbb{\mathbb{R}}^{m}$ with $n>d$. Define the function $h:\mathbb{\mathbb{R}}^{d}\to\m

Figures (1)

  • Figure 1: Runtime (in seconds) on random matrices and three real-world datasets when $p=8$ and $\varepsilon=10^{-10}$. We run each each experiment 5 times and report the mean and standard deviation of the runtime. The first row shows the absolute difference in the objectives returned by CVX and Newton's method against our algorithm on the $\log$ scale.

Theorems & Definitions (46)

  • Theorem 1.1
  • Theorem 1.2
  • Definition 2.1
  • Definition 2.2: Leverage scores
  • Definition 2.3: $\ell_{\infty}$ Lewis weights
  • Definition 2.4: $\ell_{\infty}$ Lewis weight overestimates
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3: Lemma 4.3, 4.4 adil2021unifying
  • Lemma A.1
  • ...and 36 more