The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective

TL;DR

This work analyzes the sample complexity of online reinforcement learning for unknown nonlinear dynamical systems with continuous state and action spaces in a non-episodic setting. It introduces a multi-model posterior-sampling framework that couples model identification with certainty-equivalent control, yielding nonasymptotic policy-regret guarantees across three model classes: a finite set of nonlinear candidates, a bounded function class, and a parametric family. The finite-model case achieves logarithmic regret in the horizon and model count, the infinite-class case leverages an $\epsilon$-packing to obtain a trade-off involving the packing number $m(\epsilon)$, and the parametric case achieves $O(\sqrt{N p})$ scaling with the parameter dimension $p$; all results rely on persistence of excitation and smoothness assumptions. The proposed separation principle enables offline policy computation and tractable online updates, suggesting practical applicability to large-scale nonlinear control problems and highlighting a bridge between reinforcement learning and adaptive control.

Abstract

We study the sample complexity of online reinforcement learning in the general setting of nonlinear dynamical systems with continuous state and action spaces. Our analysis accommodates a large class of dynamical systems ranging from a finite set of nonlinear candidate models to models with bounded and Lipschitz continuous dynamics, to systems that are parametrized by a compact and real-valued set of parameters. In the most general setting, our algorithm achieves a policy regret of $\mathcal{O}(N ε^2 + \mathrm{ln}(m(ε))/ε^2)$, where $N$ is the time horizon, $ε$ is a user-specified discretization width, and $m(ε)$ measures the complexity of the function class under consideration via its packing number. In the special case where the dynamics are parametrized by a compact and real-valued set of parameters (such as neural networks, transformers, etc.), we prove a policy regret of $\mathcal{O}(\sqrt{N p})$, where $p$ denotes the number of parameters, recovering earlier sample-complexity results that were derived for linear time-invariant dynamical systems. While this article focuses on characterizing sample complexity, the proposed algorithms are likely to be useful in practice, due to their simplicity, their ability to incorporate prior knowledge, and their benign transient behavior.

Figures (5)

• Figure 1: The first panel shows the evolution of the parameter error of Alg. \ref{['Alg:S1']}, while the second panel shows the evolution of the two norm of the states. The green line indicates the performance of the optimal (steady-state) policy on a different realization realization of $n_k$. We note that near-optimal steady state performance is reached in about 25 steps.
• Figure 2: The first panel shows the evolution of the parameter error of Alg. \ref{['Alg:S3']}, while the second panel shows the evolution of the two norm of the states. The green line indicates the performance of the optimal (steady-state) policy on a different realization of $n_k$. Compared to Fig. \ref{['fig:state_normS1']} the overshoot is larger and the convergence to near-optimal performance requires about 60 iterations.
• Figure 3: The first panel shows the change in policy regret of Alg. \ref{['Alg:S1']} when varying $m$. The second panel shows the evolution of the two norm of the state trajectory. We note that the behavior is consistent over the different values of $m$ (from $10$ to $10,000$, which amounts to three orders of magnitude). The green line indicates the performance of the optimal (steady-state) policy on the same realization of $n_k$.
• Figure : Reinforcement learning (S1)
• Figure : Reinforcement learning (S2)

Theorems & Definitions (16)

• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Proposition 3.1
• Theorem 3.2
• Lemma C.1
• Lemma C.2
• Lemma C.3
• Lemma C.4
• Corollary C.5