Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convex optimization with $p$-norm oracles

Deeksha Adil, Brian Bullins, Arun Jambulapati, Aaron Sidford

TL;DR

This work shows that solving ℓ_s regression can be accelerated by reducing it to smoothed ℓ_p regression for 2 ≤ p < s, achieving an iteration count of O( s · n^{ν/(1+ν)} log^2(n/ε) ) with ν = 1/p - 1/s. Central to the approach is the ℓ_p^s(λ)-proximal oracle, around which the authors build accelerated non-Euclidean proximal methods and line-search free schemes based on generalized Monteiro-Svaiter oracles. They extend the framework to non-Euclidean ball constraints and high-order smooth optimization, and establish matching lower bounds for zero-respecting algorithms, indicating near-optimality of the proposed rates. The results have potential implications for structured and higher-order optimization avenues and offer a foundation for practical speedups in regression and related convex problems.

Abstract

In recent years, there have been significant advances in efficiently solving $\ell_s$-regression using linear system solvers and $\ell_2$-regression [Adil-Kyng-Peng-Sachdeva, J. ACM'24]. Would efficient smoothed $\ell_p$-norm solvers lead to even faster rates for solving $\ell_s$-regression when $2 \leq p < s$? In this paper, we give an affirmative answer to this question and show how to solve $\ell_s$-regression using $\tilde{O}(n^{\fracν{1+ν}})$ iterations of solving smoothed $\ell_p$ regression problems, where $ν:= \frac{1}{p} - \frac{1}{s}$. To obtain this result, we provide improved accelerated rates for convex optimization problems when given access to an $\ell_p^s(λ)$-proximal oracle, which, for a point $c$, returns the solution of the regularized problem $\min_{x} f(x) + λ||x-c||_p^s$. Additionally, we show that these rates for the $\ell_p^s(λ)$-proximal oracle are optimal for algorithms that query in the span of the outputs of the oracle, and we further apply our techniques to settings of high-order and quasi-self-concordant optimization.

Convex optimization with $p$-norm oracles

TL;DR

This work shows that solving ℓ_s regression can be accelerated by reducing it to smoothed ℓ_p regression for 2 ≤ p < s, achieving an iteration count of O( s · n^{ν/(1+ν)} log^2(n/ε) ) with ν = 1/p - 1/s. Central to the approach is the ℓ_p^s(λ)-proximal oracle, around which the authors build accelerated non-Euclidean proximal methods and line-search free schemes based on generalized Monteiro-Svaiter oracles. They extend the framework to non-Euclidean ball constraints and high-order smooth optimization, and establish matching lower bounds for zero-respecting algorithms, indicating near-optimality of the proposed rates. The results have potential implications for structured and higher-order optimization avenues and offer a foundation for practical speedups in regression and related convex problems.

Abstract

In recent years, there have been significant advances in efficiently solving -regression using linear system solvers and -regression [Adil-Kyng-Peng-Sachdeva, J. ACM'24]. Would efficient smoothed -norm solvers lead to even faster rates for solving -regression when ? In this paper, we give an affirmative answer to this question and show how to solve -regression using iterations of solving smoothed regression problems, where . To obtain this result, we provide improved accelerated rates for convex optimization problems when given access to an -proximal oracle, which, for a point , returns the solution of the regularized problem . Additionally, we show that these rates for the -proximal oracle are optimal for algorithms that query in the span of the outputs of the oracle, and we further apply our techniques to settings of high-order and quasi-self-concordant optimization.

Paper Structure

This paper contains 42 sections, 40 theorems, 159 equations, 4 algorithms.

Table of Contents

  1. Introduction
  2. Efficient reductions from $\ell_s$- to $\ell_p$-regression.
  3. Optimal acceleration of $\ell_p^s(\lambda)$-proximal oracles.
  4. Optimal acceleration of non-Euclidean ball-constrained optimization oracles.
  5. Extensions to high-order optimization.
  6. Simultaneous Independent Related Work.
  7. Future work and open problems.
  8. Paper overview.
  9. Warm-up: Conceptual Proximal Point with Line Search
  10. Line-Search Free Method for Non-Euclidean Acceleration
  11. Applications
  12. $\ell_s$-Norm Regression.
  13. High-Order Smooth Optimization in General Norms.
  14. Implementing the $\ell_p$ Ball Oracle Under Hessian Stability.
  15. Lower Bounds
  16. ...and 27 more sections

Key Result

Theorem 1.2

There is an algorithm that, given $\epsilon > 0$, $s > p \geq 2$, computes an $\epsilon$-approximate solution to $\ell_s$-regression (Definition def:regression) in at most $O( s \cdot n^{\frac{\nu}{1+\nu}}\log^2\frac{n}{\epsilon})$ iterations for $\nu := \frac{1}{p} - \frac{1}{s}$, each of which can

Theorems & Definitions (78)

  • Definition 1.1: $\ell_s$-Regression
  • Theorem 1.2: From $\ell_s$-Regression to Smoothed $\ell_p$-Regression
  • Definition 1.3: $\ell_p^s(\lambda)$-Proximal Oracle
  • Theorem 1.4: Accelerated Optimization with $\ell_p^s(\lambda)$-Proximal Oracle
  • Theorem 1.5: $\ell_p^s(\lambda)$-Proximal Oracle Optimization Lower Bound
  • Definition 1.6: $\ell_p$ Ball-constrained Oracle
  • Theorem 1.7: Accelerated Optimization with $\ell_p$ Ball-constrained Oracle
  • Theorem 1.8: $\ell_p$ Ball-constrained Oracle Optimization Lower Bound
  • Theorem 1.9: High-order Optimization
  • Theorem 2.1
  • ...and 68 more