Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Convex Optimization with Unbounded Memory

Raunak Kumar, Sarah Dean, Robert Kleinberg

TL;DR

This work generalizes online convex optimization to unbounded memory, allowing the current loss to depend on the entire decision history. It introduces the $p$-effective memory capacity $H_p=( ext{sum}_{k o extinfty} k^p orm{A^k}^p)^{1/p}$ and proves matching upper and lower bounds on policy regret of order $O( ext{sqrt}{(H_p T)})$, establishing tight dependence on memory. The framework unifies and improves analyses for online linear control and online performative prediction by leveraging weighted norms and history-driven losses. The results offer constructive regret guarantees across a broad set of problems with long-term dependence, while also providing the first nontrivial lower bound for OCO with finite memory. Overall, the paper advances a flexible, broadly applicable methodology for regret minimization in sequential decision-making with unbounded history.

Abstract

Online convex optimization (OCO) is a widely used framework in online learning. In each round, the learner chooses a decision in a convex set and an adversary chooses a convex loss function, and then the learner suffers the loss associated with their current decision. However, in many applications the learner's loss depends not only on the current decision but on the entire history of decisions until that point. The OCO framework and its existing generalizations do not capture this, and they can only be applied to many settings of interest after a long series of approximation arguments. They also leave open the question of whether the dependence on memory is tight because there are no non-trivial lower bounds. In this work we introduce a generalization of the OCO framework, "Online Convex Optimization with Unbounded Memory", that captures long-term dependence on past decisions. We introduce the notion of $p$-effective memory capacity, $H_p$, that quantifies the maximum influence of past decisions on present losses. We prove an $O(\sqrt{H_p T})$ upper bound on the policy regret and a matching (worst-case) lower bound. As a special case, we prove the first non-trivial lower bound for OCO with finite memory \citep{anavaHM2015online}, which could be of independent interest, and also improve existing upper bounds. We demonstrate the broad applicability of our framework by using it to derive regret bounds, and to improve and simplify existing regret bound derivations, for a variety of online learning problems including online linear control and an online variant of performative prediction.

Online Convex Optimization with Unbounded Memory

TL;DR

This work generalizes online convex optimization to unbounded memory, allowing the current loss to depend on the entire decision history. It introduces the -effective memory capacity and proves matching upper and lower bounds on policy regret of order , establishing tight dependence on memory. The framework unifies and improves analyses for online linear control and online performative prediction by leveraging weighted norms and history-driven losses. The results offer constructive regret guarantees across a broad set of problems with long-term dependence, while also providing the first nontrivial lower bound for OCO with finite memory. Overall, the paper advances a flexible, broadly applicable methodology for regret minimization in sequential decision-making with unbounded history.

Abstract

Online convex optimization (OCO) is a widely used framework in online learning. In each round, the learner chooses a decision in a convex set and an adversary chooses a convex loss function, and then the learner suffers the loss associated with their current decision. However, in many applications the learner's loss depends not only on the current decision but on the entire history of decisions until that point. The OCO framework and its existing generalizations do not capture this, and they can only be applied to many settings of interest after a long series of approximation arguments. They also leave open the question of whether the dependence on memory is tight because there are no non-trivial lower bounds. In this work we introduce a generalization of the OCO framework, "Online Convex Optimization with Unbounded Memory", that captures long-term dependence on past decisions. We introduce the notion of -effective memory capacity, , that quantifies the maximum influence of past decisions on present losses. We prove an upper bound on the policy regret and a matching (worst-case) lower bound. As a special case, we prove the first non-trivial lower bound for OCO with finite memory \citep{anavaHM2015online}, which could be of independent interest, and also improve existing upper bounds. We demonstrate the broad applicability of our framework by using it to derive regret bounds, and to improve and simplify existing regret bound derivations, for a variety of online learning problems including online linear control and an online variant of performative prediction.
Paper Structure (44 sections, 38 theorems, 134 equations, 4 figures, 2 algorithms)

This paper contains 44 sections, 38 theorems, 134 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related work.
  4. Framework
  5. Notation.
  6. Setup
  7. Assumptions
  8. Special Cases
  9. OCO with Finite Memory.
  10. OCO with $\rho$-discounted Infinite Memory.
  11. Regret Analysis
  12. Specialization to OCO with finite memory.
  13. Comparison of upper bound with prior work.
  14. Applications
  15. Online Linear Control
  16. ...and 29 more sections

Key Result

Theorem 2.1

Consider an online convex optimization with unbounded memory problem specified by $({\cal X}, {\cal H}, A, B)$. If $f_t$ is $L$-Lipschitz continuous, then $\tilde{f}_t$ is $\tilde{L}$-Lipschitz continuous for $\tilde{L} \leq L \sum_{k=0}^\infty \| A^k \|$. If $({\cal X}, {\cal H}, A, B)$ follows lin

Figures (4)

  • Figure 1: An illustration of the loss functions $f_t$ for the OCO with finite memory lower bound.
  • Figure 2: An illustration of the loss functions $f_t$ for the OCO with finite memory lower bound. Suppose $T = 12, m = 3, L = 1$, and $p = 2$. Time is divided into blocks of size $m = 3$. Consider round $t = 5$. The history is $h_5 = (x_3, x_4, x_5)$. The loss function $f_5(h_5)$ is a product of three terms: a random sign $\epsilon_2$ sampled for the block that round $5$ belongs to, namely, block $2$; a scaling factor of $m^{-\frac{1}{2}}$; a sum over the decisions in the history excluding those that were chosen after observing $\epsilon_2$, i.e., a sum over $x_3$ and $x_4$, excluding $x_5$.
  • Figure 3: Regret plot for $\rho = 0.90$. The label OCO-UM refers to formulating the problem as an OCO with unbounded memory problem. The OCO-FM-m refers to formulating the problem as an OCO with finite memory problem with constant memory length $m$. The titles of the plots indicate the values of the dimension, the diagonal entries of $F$, and the upper triangular entries of $F$.
  • Figure 4: Regret plot for $\rho = 0.95$. The label OCO-UM refers to formulating the problem as an OCO with unbounded memory problem. The OCO-FM-m refers to formulating the problem as an OCO with finite memory problem with constant memory length $m$. The titles of the plots indicate the values of the dimension, the diagonal entries of $F$, and the upper triangular entries of $F$.

Theorems & Definitions (71)

  • Definition 2.1
  • Definition 2.2: Policy Regret
  • Definition 2.3: Linear Sequence Dynamics
  • Theorem 2.1
  • Definition 2.4: $p$-Effective Memory Capacity
  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Definition 4.1
  • ...and 61 more