Efficient Convex Optimization Requires Superlinear Memory

TL;DR

The paper establishes a fundamental memory–query tradeoff for convex optimization with a first-order oracle on the unit ball. By constructing memory-sensitive hard instances based on a Nemirovski-type function and introducing the Orthogonal Vector Game, the authors prove that any algorithm using M = d^{1.25−δ} bits of memory must perform at least \\tilde{Ω}(d^{1+4δ/3}) first-order queries to reach 1/poly(d) accuracy, revealing a polynomial gap relative to cutting-plane methods that require ~d^2 memory and achieve ~d log d queries. This result resolves a COLT 2019 open problem by Woodworth and Srebro and provides a general information-theoretic framework for memory-constrained optimization, with potential extensions to other function classes and oracle models. The work highlights that achieving near-optimal query complexity in convex optimization comes at a nontrivial memory cost, guiding the design of future algorithms and lower-bound techniques. It also offers tools—memory-sensitive bases and classes, and the Orthogonal Vector Game—that can inform broader studies of resource tradeoffs in optimization and learning.

Abstract

We show that any memory-constrained, first-order algorithm which minimizes $d$-dimensional, $1$-Lipschitz convex functions over the unit ball to $1/\mathrm{poly}(d)$ accuracy using at most $d^{1.25 - δ}$ bits of memory must make at least $\tildeΩ(d^{1 + (4/3)δ})$ first-order queries (for any constant $δ\in [0, 1/4]$). Consequently, the performance of such memory-constrained algorithms are a polynomial factor worse than the optimal $\tilde{O}(d)$ query bound for this problem obtained by cutting plane methods that use $\tilde{O}(d^2)$ memory. This resolves a COLT 2019 open problem of Woodworth and Srebro.

Figures (2)

• Figure 1: Tradeoffs between available memory and first-order oracle complexity for minimizing 1-Lipschitz convex functions over the unit ball (adapted from woodworth2019open). The dashed red region corresponds to information-theoretic lower bounds on the memory and query-complexity. The dashed green region corresponds to known upper bounds. This work shows that the solid red region is not achievable for any algorithm.
• Figure 2: A high-level overview of our proof approach. The rows of $\mathbf{A}$ and the Nemirovski vectors $\{\mathbf{v}_1,\dots, \mathbf{v}_N\}$ are sampled uniformly at random from the hypercube. $\mathbf{x}_i$ is the first query such that the oracle returns $\mathbf{v}_i$ as the response. We show that this function class is "memory-sensitive" (Definition \ref{['def:memorysensitive']}) and has the following properties: (1) to successfully minimize the function, the algorithm must see a sufficiently large number of Nemirovski vectors, (2) to reveal new Nemivoski vectors, an algorithm must make queries which are robustly linearly independent and orthogonal to $\mathbf{A}$. Using these properties, we show that minimizing the function is at least as hard as winning the Orthogonal Vector Game (Definition \ref{['def:informal_game']}) $\approx N/k$ times. Specifically, we show memory-query tradeoffs for the Orthogonal Vector Game where the goal is to obtain $k$ vectors. This tradeoff is then leveraged $\approx N/k$ times to obtain the memory-query tradeoff for the optimization problem.

Theorems & Definitions (68)

• Theorem 1
• Definition 2: Informal version of the Orthogonal Vector Game
• Definition 3: $M$-bit memory-constrained deterministic algorithm
• Definition 4: $M$-bit memory-constrained randomized algorithm
• Definition 5: Query sequence
• Definition 6: $(k$, $c_{B}$)-memory-sensitive base
• Theorem 7
• Definition 8: Induced First-Order Oracle $\mathcal{O}_{F_{f, \mathbf{A}}}$
• Definition 9: Informative subgradients
• Definition 10: $(L, N, k, \epsilon{^\star})$-memory-sensitive class