Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recursive Deep Inverse Reinforcement Learning

Paul Ghanem, Owen Howell, Michael Potter, Pau Closas, Alireza Ramezani, Deniz Erdogmus, Tales Imbiriba

TL;DR

This paper addresses online discovery of adversaries' goals by learning cost and reward functions directly from streaming demonstrations. It reformulates the maximum entropy IRL objective into a recursive, moment-matching upper bound and solves it with a Kalman-filter–inspired, second-order Newton update, enabling real-time adaptation without full trajectory batches. The proposed Recursive Deep Inverse Reinforcement Learning (RDIRL) demonstrates faster convergence and superior reward recovery across continuous-control benchmarks and a cognitive radar adversarial scenario, outperforming several leading IRL methods. The approach extends deep IRL to online settings with expressive neural-cost representations, supporting real-time counterplanning and time-critical applications.

Abstract

Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep Inverse Reinforcement Learning (IRL) methods based on maximum entropy principles show promise in recovering adversaries' goals but are typically offline, require large batch sizes with gradient descent, and rely on first-order updates, limiting their applicability in real-time scenarios. We propose an online Recursive Deep Inverse Reinforcement Learning (RDIRL) approach to recover the cost function governing the adversary actions and goals. Specifically, we minimize an upper bound on the standard Guided Cost Learning (GCL) objective using sequential second-order Newton updates, akin to the Extended Kalman Filter (EKF), leading to a fast (in terms of convergence) learning algorithm. We demonstrate that RDIRL is able to recover cost and reward functions of expert agents in standard and adversarial benchmark tasks. Experiments on benchmark tasks show that our proposed approach outperforms several leading IRL algorithms.

Recursive Deep Inverse Reinforcement Learning

TL;DR

This paper addresses online discovery of adversaries' goals by learning cost and reward functions directly from streaming demonstrations. It reformulates the maximum entropy IRL objective into a recursive, moment-matching upper bound and solves it with a Kalman-filter–inspired, second-order Newton update, enabling real-time adaptation without full trajectory batches. The proposed Recursive Deep Inverse Reinforcement Learning (RDIRL) demonstrates faster convergence and superior reward recovery across continuous-control benchmarks and a cognitive radar adversarial scenario, outperforming several leading IRL methods. The approach extends deep IRL to online settings with expressive neural-cost representations, supporting real-time counterplanning and time-critical applications.

Abstract

Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep Inverse Reinforcement Learning (IRL) methods based on maximum entropy principles show promise in recovering adversaries' goals but are typically offline, require large batch sizes with gradient descent, and rely on first-order updates, limiting their applicability in real-time scenarios. We propose an online Recursive Deep Inverse Reinforcement Learning (RDIRL) approach to recover the cost function governing the adversary actions and goals. Specifically, we minimize an upper bound on the standard Guided Cost Learning (GCL) objective using sequential second-order Newton updates, akin to the Extended Kalman Filter (EKF), leading to a fast (in terms of convergence) learning algorithm. We demonstrate that RDIRL is able to recover cost and reward functions of expert agents in standard and adversarial benchmark tasks. Experiments on benchmark tasks show that our proposed approach outperforms several leading IRL algorithms.

Paper Structure

This paper contains 18 sections, 1 theorem, 26 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. related work
  3. Background
  4. Maximum Entropy Inverse Reinforcement Learning
  5. Kalman Filtering
  6. Moment Matching as Upper Bound of the Negative Log-likelihood
  7. Recursive Deep Inverse Reinforcement Learning
  8. Experiments
  9. Continuous control
  10. Cognitive radar
  11. Conclusions
  12. Appendix
  13. Experiment details
  14. Training
  15. Reward Function and Discriminator Network Architectures
  16. ...and 3 more sections

Key Result

Theorem 5.1

Given $\hat{\theta}(t_{i-1}) \in \hat{\Theta}_{i-1}$ and known $P_{\theta_{i-1}}\in R^{d_{\theta}\times d_{\theta}}$ , the recursive equations for computing $\hat{\theta}(t_i)$ that minimizes (eq:divided_optimization_function_B) are given by the following: $P_{\theta_i}^{}$ being the lower right block of $\left(\nabla^2 \mathcal{L}_{i}(\hat{\Theta}_{i|i-1})\right)^{-1}$ recursively calculated as

Figures (4)

  • Figure 1: Learning curves for RDIRL and other methods.
  • Figure 2: Learning curves for for Walker2d.
  • Figure 3: Learning curves for RDIRL and other methods.
  • Figure 4: Learning curves for RDIRL and online adaptation methods.

Theorems & Definitions (2)

  • Theorem 5.1
  • proof