Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Computational-Query Depth Gap in Parallel Stochastic Convex Optimization

Arun Jambulapati, Aaron Sidford, Kevin Tian

TL;DR

The paper tackles parallel stochastic convex optimization by smoothing the objective with Gaussian convolution and applying a ball-acceleration framework. It introduces a Hessian-stability-based reduction that turns ball-constrained subproblems into structured stochastic quadratic problems and then solves these in parallel via a novel rank-one update maintenance strategy, leveraging a binary-search-based construction of ball oracles. The main result is a parallel algorithm that matches the best-known query depth while reducing computational depth by a polynomial factor, with complexity expressed in κ = LR/ε and ω < 2.372, and additional log factors. These advances close the depth gap in the intermediate regime and have potential implications for large-scale parallel optimization and learning systems where high parallelism is essential.

Abstract

We develop a new parallel algorithm for minimizing Lipschitz, convex functions with a stochastic subgradient oracle. The total number of queries made and the query depth, i.e., the number of parallel rounds of queries, match the prior state-of-the-art, [CJJLLST23], while improving upon the computational depth by a polynomial factor for sufficiently small accuracy. When combined with previous state-of-the-art methods our result closes a gap between the best-known query depth and the best-known computational depth of parallel algorithms. Our method starts with a ball acceleration framework of previous parallel methods, i.e., [CJJJLST20, ACJJS21], which reduce the problem to minimizing a regularized Gaussian convolution of the function constrained to Euclidean balls. By developing and leveraging new stability properties of the Hessian of this induced function, we depart from prior parallel algorithms and reduce these ball-constrained optimization problems to stochastic unconstrained quadratic minimization problems. Although we are unable to prove concentration of the asymmetric matrices that we use to approximate this Hessian, we nevertheless develop an efficient parallel method for solving these quadratics. Interestingly, our algorithms can be improved using fast matrix multiplication and use nearly-linear work if the matrix multiplication exponent is 2.

Closing the Computational-Query Depth Gap in Parallel Stochastic Convex Optimization

TL;DR

The paper tackles parallel stochastic convex optimization by smoothing the objective with Gaussian convolution and applying a ball-acceleration framework. It introduces a Hessian-stability-based reduction that turns ball-constrained subproblems into structured stochastic quadratic problems and then solves these in parallel via a novel rank-one update maintenance strategy, leveraging a binary-search-based construction of ball oracles. The main result is a parallel algorithm that matches the best-known query depth while reducing computational depth by a polynomial factor, with complexity expressed in κ = LR/ε and ω < 2.372, and additional log factors. These advances close the depth gap in the intermediate regime and have potential implications for large-scale parallel optimization and learning systems where high parallelism is essential.

Abstract

We develop a new parallel algorithm for minimizing Lipschitz, convex functions with a stochastic subgradient oracle. The total number of queries made and the query depth, i.e., the number of parallel rounds of queries, match the prior state-of-the-art, [CJJLLST23], while improving upon the computational depth by a polynomial factor for sufficiently small accuracy. When combined with previous state-of-the-art methods our result closes a gap between the best-known query depth and the best-known computational depth of parallel algorithms. Our method starts with a ball acceleration framework of previous parallel methods, i.e., [CJJJLST20, ACJJS21], which reduce the problem to minimizing a regularized Gaussian convolution of the function constrained to Euclidean balls. By developing and leveraging new stability properties of the Hessian of this induced function, we depart from prior parallel algorithms and reduce these ball-constrained optimization problems to stochastic unconstrained quadratic minimization problems. Although we are unable to prove concentration of the asymmetric matrices that we use to approximate this Hessian, we nevertheless develop an efficient parallel method for solving these quadratics. Interestingly, our algorithms can be improved using fast matrix multiplication and use nearly-linear work if the matrix multiplication exponent is 2.
Paper Structure (22 sections, 25 theorems, 110 equations, 1 table)

This paper contains 22 sections, 25 theorems, 110 equations, 1 table.

Table of Contents

  1. Introduction
  2. Our results.
  3. Paper organization.
  4. General notation.
  5. Parallel computation model.
  6. Technical overview
  7. Framework: convolutions and acceleration
  8. Gaussian convolution.
  9. Ball acceleration.
  10. Implication and approach.
  11. Hessian stability of the Gaussian convolution
  12. Hessian optimization without Hessian approximation
  13. Parallel maintenance of rank-one updates
  14. Parallel optimization of quadratic subproblems
  15. Composite stochastic optimization
  16. ...and 7 more sections

Key Result

Theorem 1

If queries to a subgradient oracle are implementable with $O(\mathcal{D}_{\mathrm{query}})$ depth and $O(\mathcal{T}_{\mathrm{query}})$ work, then there is a randomized algorithm which solves Lipschitz convex optimization with where $\omega < 2.372$AlmanDWXXZ24 is such that multiplying $d \times d$ matrices requires $O(d^\omega)$ work.

Theorems & Definitions (52)

  • Definition 1: Parallel complexity
  • Theorem 1
  • Remark 1: Parallel complexity of cutting plane methods
  • Theorem 2
  • Definition 2: Gaussian convolution
  • proof
  • Definition 3: Ball optimization oracle
  • Proposition 1: Proposition 2, CarmonJJLLST23
  • Definition 4: Matrix approximation
  • Lemma 1
  • ...and 42 more