Infinite-Horizon Reinforcement Learning with Multinomial Logistic Function Approximation
Jaehyun Park, Junyeop Kwon, Dabeen Lee
TL;DR
This work develops a provably efficient reinforcement learning algorithm, UCMNLK, for infinite-horizon MDPs with multinomial logistic (MNL) transition approximations. The approach builds confidence polytopes over transition probabilities and uses discounted extended value iteration (DEVI) to obtain optimism-based policies, achieving regret guarantees in both average-reward and discounted-reward settings. The paper establishes matching upper and lower bounds: $\tilde{\mathcal{O}}(dD\sqrt{T})$ and $\tilde{\mathcal{O}}(d(1-\gamma)^{-2}\sqrt{T})$ for the respective settings, with corresponding lower bounds $\Omega(d\sqrt{DT})$ and $\Omega(d(1-\gamma)^{3/2}\sqrt{T})$, as well as a finite-horizon bound $\Omega(dH^{3/2}\sqrt{K})$ that tightens prior results. A key technical contribution is the confidence-polytope construction, enabling tractable optimization over transition probabilities despite non-convexity in the logistic parameters, and a reduction-based technique to relate MNL transitions to linear-mixture MDPs for deriving lower bounds. The results collectively provide tight, theory-backed guarantees for RL with MNL function approximation, with potential for practical deployment in large-scale, structured RL problems where multinomial transitions are natural.
Abstract
We study model-based reinforcement learning with non-linear function approximation where the transition function of the underlying Markov decision process (MDP) is given by a multinomial logistic (MNL) model. We develop a provably efficient discounted value iteration-based algorithm that works for both infinite-horizon average-reward and discounted-reward settings. For average-reward communicating MDPs, the algorithm guarantees a regret upper bound of $\tilde{\mathcal{O}}(dD\sqrt{T})$ where $d$ is the dimension of feature mapping, $D$ is the diameter of the underlying MDP, and $T$ is the horizon. For discounted-reward MDPs, our algorithm achieves $\tilde{\mathcal{O}}(d(1-γ)^{-2}\sqrt{T})$ regret where $γ$ is the discount factor. Then we complement these upper bounds by providing several regret lower bounds. We prove a lower bound of $Ω(d\sqrt{DT})$ for learning communicating MDPs of diameter $D$ and a lower bound of $Ω(d(1-γ)^{3/2}\sqrt{T})$ for learning discounted-reward MDPs with discount factor $γ$. Lastly, we show a regret lower bound of $Ω(dH^{3/2}\sqrt{K})$ for learning $H$-horizon episodic MDPs with MNL function approximation where $K$ is the number of episodes, which improves upon the best-known lower bound for the finite-horizon setting.