Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Actuator Selection and Controller Design for Linear Quadratic Regulation with Unknown System Model

Lintao Ye, Ming Chi, Zhi-Wei Liu, Vijay Gupta

TL;DR

This work tackles online actuator selection and finite-horizon LQR controller design when the system matrices are unknown. The authors fuse a multi-armed bandit approach (Exp3.S) for selecting actuator subsets with a certainty-equivalence controller that uses online estimates of $(A,B)$ to compute time-varying LQR gains, yielding sublinear dynamic regret in episodic and non-episodic settings. They provide rigorous regret bounds that depend on problem dimensions, horizon, and the cardinality constraint, and extend the framework to large-scale instances via parallel bandit schemes and c-regret. Numerical experiments validate the theoretical results and demonstrate practical scalability for large actuator pools and high cardinality limits.

Abstract

We study the simultaneous actuator selection and controller design problem for linear quadratic regulation with Gaussian noise over a finite horizon of length $T$ and unknown system model. We consider both episodic and non-episodic settings of the problem and propose online algorithms that specify both the sets of actuators to be utilized under a cardinality constraint and the controls corresponding to the sets of selected actuators. In the episodic setting, the interaction with the system breaks into $N$ episodes, each of which restarts from a given initial condition and has length $T$. In the non-episodic setting, the interaction goes on continuously. Our online algorithms leverage a multiarmed bandit algorithm to select the sets of actuators and a certainty equivalence approach to design the corresponding controls. We show that our online algorithms yield $\sqrt{N}$-regret for the episodic setting and $T^{2/3}$-regret for the non-episodic setting. We extend our algorithm design and analysis to show scalability with respect to both the total number of candidate actuators and the cardinality constraint. We numerically validate our theoretical results.

Online Actuator Selection and Controller Design for Linear Quadratic Regulation with Unknown System Model

TL;DR

This work tackles online actuator selection and finite-horizon LQR controller design when the system matrices are unknown. The authors fuse a multi-armed bandit approach (Exp3.S) for selecting actuator subsets with a certainty-equivalence controller that uses online estimates of to compute time-varying LQR gains, yielding sublinear dynamic regret in episodic and non-episodic settings. They provide rigorous regret bounds that depend on problem dimensions, horizon, and the cardinality constraint, and extend the framework to large-scale instances via parallel bandit schemes and c-regret. Numerical experiments validate the theoretical results and demonstrate practical scalability for large actuator pools and high cardinality limits.

Abstract

We study the simultaneous actuator selection and controller design problem for linear quadratic regulation with Gaussian noise over a finite horizon of length and unknown system model. We consider both episodic and non-episodic settings of the problem and propose online algorithms that specify both the sets of actuators to be utilized under a cardinality constraint and the controls corresponding to the sets of selected actuators. In the episodic setting, the interaction with the system breaks into episodes, each of which restarts from a given initial condition and has length . In the non-episodic setting, the interaction goes on continuously. Our online algorithms leverage a multiarmed bandit algorithm to select the sets of actuators and a certainty equivalence approach to design the corresponding controls. We show that our online algorithms yield -regret for the episodic setting and -regret for the non-episodic setting. We extend our algorithm design and analysis to show scalability with respect to both the total number of candidate actuators and the cardinality constraint. We numerically validate our theoretical results.
Paper Structure (24 sections, 29 theorems, 178 equations, 4 figures)

This paper contains 24 sections, 29 theorems, 178 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Problem Formulation
  4. Online Algorithm for Episodic Setting
  5. Online Algorithm for Non-episodic Setting
  6. Algorithm Design for Episodic Setting
  7. Exp3.S Algorithm for Multiarmed Bandit
  8. Certainty Equivalence Approach
  9. Overall Algorithm Design
  10. Regret Analysis for Episodic Setting
  11. Proof of Theorem \ref{['thm:regret of as alg']}
  12. Discussions about the Results in Theorem \ref{['thm:regret of as alg']}
  13. Algorithm Design for non-episodic Setting and Regret Analysis
  14. Proof of Theorem \ref{['thm:regret of as alg non-episodic']}
  15. Handling Large-Scale Problem Instance
  16. ...and 9 more sections

Key Result

Lemma 1

auer2002nonstochastic Consider any sequence $j^{N_s}=(j_1,\dots,j_{N_s})$. In Algorithm alg:exp3.s, let $\alpha_2=1/{N_s}$ and Let $\mathbb{E}_M[\cdot]$ denote the expectation with respective to the randomness in the algorithm. Then, we have

Figures (4)

  • Figure 1: $R_{\mathcal{A}_e}/N$ and $R_{\mathcal{A}_e}/\sqrt{N}$ against $N$.
  • Figure 2: The influence of the problem parameter $H$ (resp., $n$) on the performance (resp., running times) of Algorithm \ref{['alg:online for AS']}.
  • Figure 3: $R_{\mathcal{A}_l^{\prime}}/T$ and $R_{\mathcal{A}_l^{\prime}}/T^{3/4}$ against $T$.
  • Figure 4: The running times of Algorithm \ref{['alg:online for AS large scale']} against $m$ and $H$.

Theorems & Definitions (34)

  • Remark 1
  • Lemma 1
  • Remark 2
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Proposition 1
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • ...and 24 more