Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models
Lars van der Laan, Aurelien Bibaut, Nathan Kallus
TL;DR
This work develops a semiparametric framework for inference in softmax-based inverse reinforcement learning (IRL) and dynamic discrete choice (DDC) models with nonparametric rewards. By showing that the log-behavior policy $r_0=\log\pi_0$ can act as a pseudo-reward and that a normalization constraint yields point identification of the true reward, the authors derive efficient influence functions and construct automatic debiased machine-learning estimators (autoDML) that achieve $\sqrt{n}$-consistency and semiparametric efficiency. The framework expresses reward-dependent estimands as smooth functionals of observable quantities (behavior policy and transition kernel), enabling robust policy evaluation without nested dynamic programming or restrictive parametric assumptions. This unifies MaxEnt IRL and Gumbel-shock DDC, provides practical inference tools using standard ML and fitted $Q$-iteration, and offers a template for extending to broader shock distributions and nonstationary settings with principled efficiency guarantees.
Abstract
Inverse reinforcement learning (IRL) and dynamic discrete choice (DDC) models explain sequential decision-making by recovering reward functions that rationalize observed behavior. Flexible IRL methods typically rely on machine learning but provide no guarantees for valid inference, while classical DDC approaches impose restrictive parametric specifications and often require repeated dynamic programming. We develop a semiparametric framework for debiased inverse reinforcement learning that yields statistically efficient inference for a broad class of reward-dependent functionals in maximum entropy IRL and Gumbel-shock DDC models. We show that the log-behavior policy acts as a pseudo-reward that point-identifies policy value differences and, under a simple normalization, the reward itself. We then formalize these targets, including policy values under known and counterfactual softmax policies and functionals of the normalized reward, as smooth functionals of the behavior policy and transition kernel, establish pathwise differentiability, and derive their efficient influence functions. Building on this characterization, we construct automatic debiased machine-learning estimators that allow flexible nonparametric estimation of nuisance components while achieving $\sqrt{n}$-consistency, asymptotic normality, and semiparametric efficiency. Our framework extends classical inference for DDC models to nonparametric rewards and modern machine-learning tools, providing a unified and computationally tractable approach to statistical inference in IRL.