Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
Denis Belomestny, Alexey Naumov, Sergey Samsonov
TL;DR
This work develops a rigorous statistical theory for inverse entropy-regularized reinforcement learning (ER-IRL) by coupling penalized maximum-likelihood policy estimation with a least-squares reward reconstruction. It provides non-asymptotic, high-probability bounds for the excess policy KL divergence and the LS-recovered reward, and establishes minimax-optimal convergence rates through matching lower bounds. The analysis reveals how entropy regularization, model complexity (via covering numbers), and sample size jointly govern reward identifiability and estimation accuracy, bridging behavior cloning and IRL within a modern statistical learning framework. Extensions include basis projections, Bayesian interpretations, and connections to contextual Bradley–Terry models, offering a comprehensive toolkit for robust reward recovery in high-dimensional settings.
Abstract
Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy $π^\star$ and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.