Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Bandit Nonlinear Control with Dynamic Batch Length and Adaptive Learning Rate

Jihun Kim, Javad Lavaei

TL;DR

Dynamic batch length and adaptive learning rate in DBAR enables the online bandit nonlinear control system to attain asymptotic stability, where the algorithm behaves as if there were no destabilizing controllers.

Abstract

This paper is concerned with the online bandit nonlinear control, which aims to learn the best stabilizing controller from a pool of stabilizing and destabilizing controllers of unknown types for a given nonlinear dynamical system. We develop an algorithm, named Dynamic Batch length and Adaptive learning Rate (DBAR), and study its stability and regret. Unlike the existing Exp3 algorithm requiring an exponentially stabilizing controller, DBAR only needs a significantly weaker notion of controller stability, in which case substantial time may be required to certify the system stability. Dynamic batch length in DBAR effectively addresses this issue and enables the system to attain asymptotic stability, where the algorithm behaves as if there were no destabilizing controllers. Moreover, adaptive learning rate in DBAR only uses the state norm information to achieve a tight regret bound even when none of the stabilizing controllers in the pool are exponentially stabilizing.

Online Bandit Nonlinear Control with Dynamic Batch Length and Adaptive Learning Rate

TL;DR

Dynamic batch length and adaptive learning rate in DBAR enables the online bandit nonlinear control system to attain asymptotic stability, where the algorithm behaves as if there were no destabilizing controllers.

Abstract

This paper is concerned with the online bandit nonlinear control, which aims to learn the best stabilizing controller from a pool of stabilizing and destabilizing controllers of unknown types for a given nonlinear dynamical system. We develop an algorithm, named Dynamic Batch length and Adaptive learning Rate (DBAR), and study its stability and regret. Unlike the existing Exp3 algorithm requiring an exponentially stabilizing controller, DBAR only needs a significantly weaker notion of controller stability, in which case substantial time may be required to certify the system stability. Dynamic batch length in DBAR effectively addresses this issue and enables the system to attain asymptotic stability, where the algorithm behaves as if there were no destabilizing controllers. Moreover, adaptive learning rate in DBAR only uses the state norm information to achieve a tight regret bound even when none of the stabilizing controllers in the pool are exponentially stabilizing.
Paper Structure (17 sections, 31 theorems, 132 equations, 7 figures, 2 tables, 3 algorithms)

This paper contains 17 sections, 31 theorems, 132 equations, 7 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Algorithm Description
  4. Main Results
  5. Stability
  6. Regret
  7. Numerical Experiments
  8. Conclusion
  9. Necessity of DBAR under weaker stability notion of required controllers
  10. Glossary
  11. Stability Proof
  12. Regret Proof for Algorithm \ref{['Algorithm 1']}
  13. Regret Proof for Algorithm \ref{['Algorithm 2']}
  14. Applications: Switched systems
  15. Numerical Experiment Details
  16. ...and 2 more sections

Key Result

Theorem 4.1

In Algorithm Algorithm 1, suppose that $\frac{\tau_1}{\tau_0} \beta(\tau_0)<1$. Then, it holds that $\lim_{T\to\infty}\frac{1}{T}\sum_{t=0}^T \|x_t\| \leq \gamma w_\text{max}.$

Figures (7)

  • Figure 1: The stability and the regret in the linear system under sinusoidal noise. Fixed $\tau$, fixed $\eta$ represents the algorithm in li2023switching. Ablation study of the algorithm is presented.
  • Figure 2: The stability and the regret in the ball-beam system under sinusoidal noise. We selected $\beta(t) = \min\{10/t,1\}$ (see Definition \ref{['iss']}) and used squared sum of state and action norms as the cost.
  • Figure 3: The state norm with a fixed batch length compared to that with a dynamic batch length. $x_{t+1}=x_t+0.15u_t+w_t$ with $u_t=Kx_t$ where $K\in [-3.0,-2.9,-2.8, \dots, 4.9, 5.0]$. We use $\tau_0=10, \gamma=3$, and set $w_\text{max}=0.5$. The noise $w_t$ is (a) i.i.d. sampled from Uniform$[-0.2, 0.5]$, and (b) $0.15+0.35\sin(\frac{t}{3\pi})$.
  • Figure 4: The stability and the regret in the linear system under truncated Gaussian noise. Ablation study of the algorithm is presented.
  • Figure 5: The stability and the regret in the linear system under Uniform random walk. Ablation study of the algorithm is presented.
  • ...and 2 more figures

Theorems & Definitions (68)

  • Definition 2.3: Input-to-state stable controller
  • Definition 2.4: Incrementally stable controller
  • Definition 2.6: Stabilizing and destabilizing controller
  • Remark 2.7
  • Definition 2.8: Asymptotic stability
  • Definition 2.9: Finite-gain stability
  • Definition 2.10: Regret
  • Remark 3.2
  • Theorem 4.1: Asymptotic stability
  • Theorem 4.2: Finite-gain stability
  • ...and 58 more