Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimistic Online Non-stochastic Control via FTRL

Naram Mhaisen, George Iosifidis

TL;DR

This work extends online non-stochastic control by introducing optimistic learning with a prediction oracle into Disturbance Action Controllers (DAC). The proposed OptFTRL-C algorithm treats NSC with memory as an online learning problem with delayed feedback, deriving a bound on policy regret that scales with prediction accuracy and memory length. By decomposing costs into separable, delayed components and leveraging optimistic delay frameworks, the authors achieve regret guarantees ranging from $\mathcal{O}(1)$ for perfect predictions to $\mathcal{O}(\sqrt{T})$ in the worst case, while remaining robust to untrusted forecasts. The approach enables learning-based controllers that effectively exploit good predictions without sacrificing resilience when forecasts are poor, advancing practical NSC with data-driven predictions.

Abstract

This paper brings the concept of ``optimism" to the new and promising framework of online Non-stochastic Control (NSC). Namely, we study how NSC can benefit from a prediction oracle of unknown quality responsible for forecasting future costs. The posed problem is first reduced to an optimistic learning with delayed feedback problem, which is handled through the Optimistic Follow the Regularized Leader (OFTRL) algorithmic family. This reduction enables the design of \texttt{OptFTRL-C}, the first Disturbance Action Controller (DAC) with optimistic policy regret bounds. These new bounds are commensurate with the oracle's accuracy, ranging from $\mathcal{O}(1)$ for perfect predictions to the order-optimal $\mathcal{O}(\sqrt{T})$ even when all predictions fail. By addressing the challenge of incorporating untrusted predictions into online control, this work contributes to the advancement of the NSC framework and paves the way toward effective and robust learning-based controllers.

Optimistic Online Non-stochastic Control via FTRL

TL;DR

This work extends online non-stochastic control by introducing optimistic learning with a prediction oracle into Disturbance Action Controllers (DAC). The proposed OptFTRL-C algorithm treats NSC with memory as an online learning problem with delayed feedback, deriving a bound on policy regret that scales with prediction accuracy and memory length. By decomposing costs into separable, delayed components and leveraging optimistic delay frameworks, the authors achieve regret guarantees ranging from for perfect predictions to in the worst case, while remaining robust to untrusted forecasts. The approach enables learning-based controllers that effectively exploit good predictions without sacrificing resilience when forecasts are poor, advancing practical NSC with data-driven predictions.

Abstract

This paper brings the concept of ``optimism" to the new and promising framework of online Non-stochastic Control (NSC). Namely, we study how NSC can benefit from a prediction oracle of unknown quality responsible for forecasting future costs. The posed problem is first reduced to an optimistic learning with delayed feedback problem, which is handled through the Optimistic Follow the Regularized Leader (OFTRL) algorithmic family. This reduction enables the design of \texttt{OptFTRL-C}, the first Disturbance Action Controller (DAC) with optimistic policy regret bounds. These new bounds are commensurate with the oracle's accuracy, ranging from for perfect predictions to the order-optimal even when all predictions fail. By addressing the challenge of incorporating untrusted predictions into online control, this work contributes to the advancement of the NSC framework and paves the way toward effective and robust learning-based controllers.
Paper Structure (11 sections, 4 theorems, 26 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 11 sections, 4 theorems, 26 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Contributions
  4. Related Work
  5. Preliminaries
  6. Online control with Optimistic FTRL
  7. NSC with linearized costs is separable OCO-M.
  8. separable OCO-M is OCO-D
  9. Numerical Example
  10. Conclusion
  11. Acknowledgment

Key Result

Theorem 1

Let $(A,B)$ be an LTI system, and $\{c_t(\cdot,\cdot)\}_{t=1}^T$, $\{\boldsymbol{w}\}_{t=1}^T$ be any sequence of stage costs and disturbances, respectively. Let $\Delta^{(i)}_t$ be the prediction error at $t$ with attenuation level $i$, as defined in eq:deltai. Then, with memory parameter $d$ defin

Figures (2)

  • Figure 1: The methodology of designing OptFTRL-C. Up: The NSC to OCO-M reduction, with parameter $d=2$ (Sec. \ref{['subsec:A']}). Down: an equivalent delayed OCO formulation, obtained via rearrangement, which we append with an oracle (Sec. \ref{['subsec:B']}).
  • Figure 2: The average regret against the optimal policy under various scenarios (cost and disturbances trajectories).

Theorems & Definitions (7)

  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • proof
  • Theorem 2: optimdelay21