Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control

Shivam Bajaj, Prateek Jaiswal, Vijay Gupta

TL;DR

This work tackles the sim2real gap in online linear quadratic control by leveraging offline data from a similar, but unknown, system. It introduces a Thompson Sampling-based algorithm (TSOD-LQR) that fuses offline trajectory information with online observations, providing Bayesian regret guarantees that scale as $\tilde{\mathcal{O}}(f(S,M_{\delta})\sqrt{T/S})$, where $S$ is the offline trajectory length and $M_{\delta}$ bounds system dissimilarity. Theoretical results show improved regret when the offline source is similar ($M_{\delta}$ small) and demonstrate extensions to multiple offline sources. Numerical experiments corroborate the benefits of incorporating offline data, with regret decreasing as $S$ grows and outperformance over naive online-only strategies. The framework offers a practical route for safer, faster online adaptation in real-world systems by exploiting available simulations or simpler models.

Abstract

``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length $S$ of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length $S$ is obtained. We establish that the algorithm achieves $\tilde{\mathcal{O}}({f(S,M_δ)\sqrt{T/S}})$ Bayes regret after $T$ time steps, where $M_δ$ characterizes the \emph{dissimilarity} between the two systems and $f(S,M_δ)$ is a function of $S$ and $M_δ$. When $M_δ$ is sufficiently small, the proposed algorithm achieves $\tilde{\mathcal{O}}({\sqrt{T/S}})$ Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control

TL;DR

This work tackles the sim2real gap in online linear quadratic control by leveraging offline data from a similar, but unknown, system. It introduces a Thompson Sampling-based algorithm (TSOD-LQR) that fuses offline trajectory information with online observations, providing Bayesian regret guarantees that scale as , where is the offline trajectory length and bounds system dissimilarity. Theoretical results show improved regret when the offline source is similar ( small) and demonstrate extensions to multiple offline sources. Numerical experiments corroborate the benefits of incorporating offline data, with regret decreasing as grows and outperformance over naive online-only strategies. The framework offers a practical route for safer, faster online adaptation in real-world systems by exploiting available simulations or simpler models.

Abstract

``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length is obtained. We establish that the algorithm achieves Bayes regret after time steps, where characterizes the \emph{dissimilarity} between the two systems and is a function of and . When is sufficiently small, the proposed algorithm achieves Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.
Paper Structure (15 sections, 17 theorems, 53 equations, 2 figures, 1 algorithm)

This paper contains 15 sections, 17 theorems, 53 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Classical LQR Design
  4. Model and Problem Statement
  5. Offline Data-Generation
  6. Thompson Sampling with Offline Data for LQR (TSOD-LQR) Algorithm
  7. Regret Analysis
  8. Numerical Results
  9. Extension to Multiple Offline Sources
  10. Conclusion
  11. Acknowledgement
  12. Proof of Theorem \ref{['thm:online_bound']}
  13. Proof of Lemma \ref{['lem:R_2']}
  14. Proof of Lemma \ref{['lem:R_3']}
  15. Additional Lemmas

Key Result

Lemma 1

The set $\mathcal{Q}$ is compact. For any $\theta\in \mathcal{Q}$, $\theta$ is stabilizable and there exists a constant $M_K< \infty$, where $M_K\coloneqq \sup_{\theta\in\mathcal{Q}}\|K(\theta)\|_2$.

Figures (2)

  • Figure 1: Cumulative regret plot comparing Algorithm \ref{['algo:TS_LQR']} with an algorithm that (1) does not utilize the offline data and (2) only utilizes the estimate $\hat{\theta}^{\text{sim}}$ computed from the offline data.
  • Figure 2: Cumulative regret plot comparing Algorithm \ref{['algo:TS_LQR']} for various values of $S$ and $M_{\delta} = 0.15$.

Theorems & Definitions (32)

  • Lemma 1: Proposition 5 in abeille2018improved
  • Theorem 4.1
  • Lemma 2
  • proof
  • Remark 1
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • ...and 22 more