Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-Based Reinforcement Learning with Multinomial Logistic Function Approximation

Taehyun Hwang, Min-hwan Oh

TL;DR

This paper proposes a provably efficient RL algorithm for the MDP whose state transition is given by a multinomial logistic model and shows that it consistently outperforms the existing methods, hence achieving both provable efficiency and practical superior performance.

Abstract

We study model-based reinforcement learning (RL) for episodic Markov decision processes (MDP) whose transition probability is parametrized by an unknown transition core with features of state and action. Despite much recent progress in analyzing algorithms in the linear MDP setting, the understanding of more general transition models is very restrictive. In this paper, we establish a provably efficient RL algorithm for the MDP whose state transition is given by a multinomial logistic model. To balance the exploration-exploitation trade-off, we propose an upper confidence bound-based algorithm. We show that our proposed algorithm achieves $\tilde{O}(d \sqrt{H^3 T})$ regret bound where $d$ is the dimension of the transition core, $H$ is the horizon, and $T$ is the total number of steps. To the best of our knowledge, this is the first model-based RL algorithm with multinomial logistic function approximation with provable guarantees. We also comprehensively evaluate our proposed algorithm numerically and show that it consistently outperforms the existing methods, hence achieving both provable efficiency and practical superior performance.

Model-Based Reinforcement Learning with Multinomial Logistic Function Approximation

TL;DR

This paper proposes a provably efficient RL algorithm for the MDP whose state transition is given by a multinomial logistic model and shows that it consistently outperforms the existing methods, hence achieving both provable efficiency and practical superior performance.

Abstract

We study model-based reinforcement learning (RL) for episodic Markov decision processes (MDP) whose transition probability is parametrized by an unknown transition core with features of state and action. Despite much recent progress in analyzing algorithms in the linear MDP setting, the understanding of more general transition models is very restrictive. In this paper, we establish a provably efficient RL algorithm for the MDP whose state transition is given by a multinomial logistic model. To balance the exploration-exploitation trade-off, we propose an upper confidence bound-based algorithm. We show that our proposed algorithm achieves regret bound where is the dimension of the transition core, is the horizon, and is the total number of steps. To the best of our knowledge, this is the first model-based RL algorithm with multinomial logistic function approximation with provable guarantees. We also comprehensively evaluate our proposed algorithm numerically and show that it consistently outperforms the existing methods, hence achieving both provable efficiency and practical superior performance.
Paper Structure (28 sections, 12 theorems, 76 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 28 sections, 12 theorems, 76 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Notations
  5. Problem Formulation
  6. Linear Transition Model vs. Multinomial Logistic Transition Model
  7. Algorithm and Main Results
  8. UCRL-MNL Algorithm
  9. Estimation of Transition Core
  10. Model-Based Upper Confidence
  11. Closed-Form UCB
  12. Regret Bound for UCRL-MNL Algorithm
  13. Discussion of Theorem \ref{['thm_main theorem']}
  14. Proof Sketch and Key Lemmas
  15. Numerical Experiments
  16. ...and 13 more sections

Key Result

Proposition 1

For an arbitrary set of features of state and actions of an MDP, there exist no linear transition model that can induce a proper probability distribution over next states.

Figures (3)

  • Figure 1: The "RiverSwim" environment with $n$ states osband2013more
  • Figure 2: Episodic returns over 10 independent runs under the different RiverSwim environments
  • Figure 3: Episodic returns over 10 independent runs under the different RiverSwim environments (without smoothing effect)

Theorems & Definitions (14)

  • Definition 1: Reachable states
  • Remark 1
  • Proposition 1
  • Proposition 2
  • Theorem 1: Regret bound of $\texttt{UCRL-MNL}$
  • Lemma 1: Concentration of the transition core
  • Lemma 2: Optimism
  • Lemma 3: Concentration of the value function
  • Lemma 4: Lemma 1 in li2024improved
  • Lemma 5: Good event
  • ...and 4 more