Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge
Meshal Alharbi, Mardavij Roozbehani, Munther Dahleh
TL;DR
This work investigates sample-efficient online reinforcement learning when partial knowledge of system dynamics is available, focusing on additive disturbance dynamics $S_{h+1}=f(S_h,A_h)+W_h$. It introduces an optimistic Q-learning algorithm that exploits a known or learnable dynamics function $f$ to achieve fast regret scaling, including $\tilde{\mathcal{O}}(\sqrt{H^6 T})$ when $f$ is known and a sample complexity independent of $|\mathcal{S}|$ and $|\mathcal{A}|$ when a noisy $\hat{f}$ is available with bounded error $\zeta$. The paper further shows that if an online estimator $\{\hat{f}_i\}$ converges at rate $\|\hat{f}_i-f\|_\infty=\mathcal{O}(\sqrt{d/i})$, the regret becomes $\tilde{O}(\sqrt{H^6 T}+L\sqrt{HdT})$, tying learning efficiency to the complexity of learning $f$. Not requiring explicit transition-modeling or DP solvers, the approach retains model-free memory characteristics and demonstrates empirical advantages over problem-agnostic baselines, with practical implications for control and operations research where partial dynamics are known or learnable.
Abstract
The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form $S_{h+1} = f(S_h, A_h) + W_h$, where $f$ represents the underlying system dynamics, and $W_h$ are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with $S$ states, $A$ actions, and episode length $H$, we present an optimistic Q-learning algorithm that achieves $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ regret under perfect knowledge of $f$, where $T$ is the total number of interactions with the system. This is in contrast to the typical $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ regret for existing Q-learning methods. Further, if only a noisy estimate $\hat{f}$ of $f$ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error $\hat{f}-f$, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.