Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimality
Ruijia Zhang, Siliang Zeng, Chenliang Li, Alfredo Garcia, Mingyi Hong
TL;DR
This work addresses inverse reinforcement learning with neural network–parameterized rewards and proposes a two-timescale, single-loop ML-IRL framework that uses a dynamically truncated neural soft Q-learning inner loop. It provides non-asymptotic convergence guarantees and, under overparameterization, conditions for global optimality of the learned reward and the corresponding policy, with finite-time complexity characterized by $\mathcal{O}(\epsilon^{-8})$ steps to achieve $\epsilon$-optimality. The analysis relies on local linearization of the neural networks for both the Q-function and the reward, along with ergodicity and regularity assumptions, yielding concrete rates for policy and gradient convergence. Empirically, the method demonstrates superior imitation performance on Mujoco tasks with a single expert trajectory, validating the practical impact of scalable neural-reward IRL with provable guarantees. Overall, the paper advances theoretical understanding and practical application of neural-network–parameterized IRL using a provably efficient, single-loop optimization framework.
Abstract
The goal of the Inverse reinforcement learning (IRL) task is to identify the underlying reward function and the corresponding optimal policy from a set of expert demonstrations. While most IRL algorithms' theoretical guarantees rely on a linear reward structure, we aim to extend the theoretical understanding of IRL to scenarios where the reward function is parameterized by neural networks. Meanwhile, conventional IRL algorithms usually adopt a nested structure, leading to computational inefficiency, especially in high-dimensional settings. To address this problem, we propose the first two-timescale single-loop IRL algorithm under neural network parameterized reward and provide a non-asymptotic convergence analysis under overparameterization. Although prior optimality results for linear rewards do not apply, we show that our algorithm can identify the globally optimal reward and policy under certain neural network structures. This is the first IRL algorithm with a non-asymptotic convergence guarantee that provably achieves global optimality in neural network settings.