Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thompson Sampling-Based Learning and Control for Unknown Dynamic Systems

Kaikai Zheng, Dawei Shi, Yang Shi, Long Wang

TL;DR

This paper addresses learning-based control for unknown nonlinear dynamics by combining Thompson Sampling with a Hilbert-space (RKHS) parameterization of controllers. It constructs a convex, RKHS-backed function space around an initial controller and uses TS to explore and exploit candidate control laws, with a Bayesian posterior update for the unknown cost function. The authors prove exponential convergence of the learned cost to a neighborhood of the true cost, derive a finite upper bound on control regret that separates a stationary approximation error from a decaying exploration term, and analyze closed-loop mean-square stability under the learning process. Extensions include adaptive segment lengths, nonstationary reward handling, and stability guarantees, with computational complexity characterized for practical kernel-based implementations. Numerical simulations on unknown nonlinear systems validate rapid convergence, meaningful regret bounds, and favorable performance compared to GP-MPC, highlighting the approach’s robustness and scalability for data-driven control without explicit system identification.

Abstract

Thompson sampling (TS) is a Bayesian randomized exploration strategy that samples options (e.g., system parameters or control laws) from the current posterior and then applies the selected option that is optimal for a task, thereby balancing exploration and exploitation; this makes TS effective for active learning-based controller design. However, TS relies on finite parametric representations, which limits its applicability to more general spaces, which are more commonly encountered in control system design. To address this issue, this work proposes a parameterization method for control law learning using reproducing kernel Hilbert spaces and designs a data-driven active learning control approach. Specifically, the proposed method treats the control law as an element in a function space, allowing the design of control laws without imposing restrictions on the system structure or the form of the controller. A TS framework is proposed in this work to reduce control costs through online exploration and exploitation, and the convergence guarantees are further provided for the learning process. Theoretical analysis shows that the proposed method learns the relationship between control laws and closed-loop performance metrics at an exponential rate, and the upper bound of control regret is also derived. Furthermore, the closed-loop stability of the proposed learning framework is analyzed. Numerical experiments on controlling unknown nonlinear systems validate the effectiveness of the proposed method.

Thompson Sampling-Based Learning and Control for Unknown Dynamic Systems

TL;DR

This paper addresses learning-based control for unknown nonlinear dynamics by combining Thompson Sampling with a Hilbert-space (RKHS) parameterization of controllers. It constructs a convex, RKHS-backed function space around an initial controller and uses TS to explore and exploit candidate control laws, with a Bayesian posterior update for the unknown cost function. The authors prove exponential convergence of the learned cost to a neighborhood of the true cost, derive a finite upper bound on control regret that separates a stationary approximation error from a decaying exploration term, and analyze closed-loop mean-square stability under the learning process. Extensions include adaptive segment lengths, nonstationary reward handling, and stability guarantees, with computational complexity characterized for practical kernel-based implementations. Numerical simulations on unknown nonlinear systems validate rapid convergence, meaningful regret bounds, and favorable performance compared to GP-MPC, highlighting the approach’s robustness and scalability for data-driven control without explicit system identification.

Abstract

Thompson sampling (TS) is a Bayesian randomized exploration strategy that samples options (e.g., system parameters or control laws) from the current posterior and then applies the selected option that is optimal for a task, thereby balancing exploration and exploitation; this makes TS effective for active learning-based controller design. However, TS relies on finite parametric representations, which limits its applicability to more general spaces, which are more commonly encountered in control system design. To address this issue, this work proposes a parameterization method for control law learning using reproducing kernel Hilbert spaces and designs a data-driven active learning control approach. Specifically, the proposed method treats the control law as an element in a function space, allowing the design of control laws without imposing restrictions on the system structure or the form of the controller. A TS framework is proposed in this work to reduce control costs through online exploration and exploitation, and the convergence guarantees are further provided for the learning process. Theoretical analysis shows that the proposed method learns the relationship between control laws and closed-loop performance metrics at an exponential rate, and the upper bound of control regret is also derived. Furthermore, the closed-loop stability of the proposed learning framework is analyzed. Numerical experiments on controlling unknown nonlinear systems validate the effectiveness of the proposed method.

Paper Structure

This paper contains 19 sections, 10 theorems, 118 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Preliminaries
  4. Neighborhood
  5. Thompson Sampling
  6. Main Results
  7. Parameterization of Controllers
  8. TS-based Learning Algorithm
  9. Convergence Analysis of the Learning Process
  10. Analysis of Control Regret
  11. Extensions and Closed-Loop Stability Analysis
  12. Extensions on Information Gain-Based Sampling Stopping Mechanism and Unknown Initial State Distribution
  13. Modification on the Sampling Stopping Criterion
  14. Corollaries of Learning Convergence and Control Regret
  15. Extension on Nonstationary Reward Distribution
  16. ...and 4 more sections

Key Result

Proposition 1

If Assumption assum:Xbar holds, $\forall {\bm x},\tilde{\bm x}\in\mathcal{X}$ and $\forall {\bm w}\subseteq[1,n]$, $({\bm x},\tilde{\bm x})_{\bm w}\in\bar{\mathcal{X}}$.

Figures (5)

  • Figure 1: Basis functions of the function space $\mathcal{G}$, which is obtained from the parameterization approach \ref{['eq:calGi']}.
  • Figure 2: Hellinger distance (top subplot) and KL divergence (bottom subplot) between the learned reward function $\bar{J}^t$ and the ground truth $\bar{J}^*$.
  • Figure 3: Convergence of the learned reward function $\bar{J}^t$ as a function of time. The blue line represents the average value, while the shaded area with dashed lines represents the standard deviation deviating from the average value.
  • Figure 4: Rewards during the proposed ALC processes. The red solid line represents the average of 500 independent trials, the light red area indicates the range within one standard deviation from the mean, and the blue dashed line represents the reward of the optimal control law under noise-free conditions.
  • Figure 5: The convergence process in a single test. The black hexagram represents the control law with maximized reward (namely, $\bar{\bm g}$), and the solid dots represent individual samples. The color of each dot indicates the sampling order: blue dots correspond to samples taken at the beginning of the learning process, green and yellow dots represent samples after some exploration, and red dots indicate samples from the 150th to the 200th segments.

Theorems & Definitions (31)

  • Remark 1
  • Definition 1
  • Definition 2
  • Definition 3
  • Remark 2
  • Proposition 1
  • Proof
  • Example 1
  • Remark 3
  • Lemma 1
  • ...and 21 more