Online Reinforcement Learning in Markov Decision Process Using Linear Programming

TL;DR

The proposed algorithm, which is based on the concept of “optimism in the face of uncertainty”, maintains confidence sets of transition and reward functions and uses occupancy measures to connect the online MDP with linear programming to achieve a tighter regret bound.

Abstract

We consider online reinforcement learning in episodic Markov decision process (MDP) with unknown transition function and stochastic rewards drawn from some fixed but unknown distribution. The learner aims to learn the optimal policy and minimize their regret over a finite time horizon through interacting with the environment. We devise a simple and efficient model-based algorithm that achieves $\widetilde{O}(LX\sqrt{TA})$ regret with high probability, where $L$ is the episode length, $T$ is the number of episodes, and $X$ and $A$ are the cardinalities of the state space and the action space, respectively. The proposed algorithm, which is based on the concept of ``optimism in the face of uncertainty", maintains confidence sets of transition and reward functions and uses occupancy measures to connect the online MDP with linear programming. It achieves a tighter regret bound compared to the existing works that use a similar confidence set framework and improves computational effort compared to those that use a different framework but with a slightly tighter regret bound.

Figures (1)

• Figure 1: Performance of UCRL-LP and UCRL2 algorithms.

Theorems & Definitions (21)

• Remark 1
• Definition 1
• Proposition 1: rosenberg2019onlineconvexrosenberg2019onlinestochasticjin2020learning
• Proposition 2: rosenberg2019onlineconvexrosenberg2019onlinestochasticjin2020learning
• Proposition 3: altman1999bookrosenberg2019onlineconvexrosenberg2019onlinestochasticjin2020learning
• Lemma 1: Theorem 4.2 in rosenberg2019onlineconvex
• Lemma 2: Lemma 4.1 in rosenberg2019onlineconvex
• Lemma 3
• Theorem 4
• proof