Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
Titouan Renard, Andreas Schlaginhaufen, Tingting Ni, Maryam Kamgarpour
TL;DR
Inverse reinforcement learning is cast as recovering a reward that rationalizes expert behavior in an unknown MDP. The paper proposes a model-free, single-loop entropy-regularized IRL algorithm that jointly updates the reward via stochastic projected gradient descent and the policy via stochastic soft policy iteration, using a generative model for unbiased estimation. It provides end-to-end guarantees: the recovered reward is $\varepsilon$-optimal for the expert with $\mathcal{O}(1/\varepsilon^{2})$ samples, and the optimal policy under that reward is $\varepsilon$-close to the expert in total variation with $\mathcal{O}(1/\varepsilon^{4})$ samples, under standard realizability and exploration assumptions. These results advance theoretical understanding of model-free IRL with entropy regularization by delivering concrete, finite-sample guarantees for both reward recovery and policy proximity.
Abstract
Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is $\varepsilon$-optimal using $\mathcal{O}(1/\varepsilon^{2})$ samples of the Markov decision process (MDP). Furthermore, with $\mathcal{O}(1/\varepsilon^{4})$ samples we prove that the optimal policy corresponding to the recovered reward is $\varepsilon$-close to the expert policy in total variation distance.