A Differential Dynamic Programming Framework for Inverse Reinforcement Learning

TL;DR

We address the inverse reinforcement learning (IRL) problem under general nonlinear, constrained dynamics by developing a differential dynamic programming ($DDP$) framework that can recover cost, dynamics, and constraint parameters from demonstrations. The inner loop solves a constrained OC problem with an augmented interior-point Bellman equation, enabling an efficient one-step gradient of the outer objective with respect to $m{ heta}$; this gradient is shown to be equivalent to differentiating the PMP conditions ($PMP$). A closed-loop IRL loss is proposed to capture the feedback nature of real demonstrations, leading to improved recoverability under certain rank conditions. The framework is validated on four numerical systems and a real quadrotor, demonstrating the practical benefits of closed-loop IRL over open-loop IRL and confirming the theoretical recoverability results.

Abstract

A differential dynamic programming (DDP)-based framework for inverse reinforcement learning (IRL) is introduced to recover the parameters in the cost function, system dynamics, and constraints from demonstrations. Different from existing work, where DDP was used for the inner forward problem with inequality constraints, our proposed framework uses it for efficient computation of the gradient required in the outer inverse problem with equality and inequality constraints. The equivalence between the proposed method and existing methods based on Pontryagin's Maximum Principle (PMP) is established. More importantly, using this DDP-based IRL with an open-loop loss function, a closed-loop IRL framework is presented. In this framework, a loss function is proposed to capture the closed-loop nature of demonstrations. It is shown to be better than the commonly used open-loop loss function. We show that the closed-loop IRL framework reduces to a constrained inverse optimal control problem under certain assumptions. Under these assumptions and a rank condition, it is proven that the learning parameters can be recovered from the demonstration data. The proposed framework is extensively evaluated through four numerical robot examples and one real-world quadrotor system. The experiments validate the theoretical results and illustrate the practical relevance of the approach.

Figures (15)

• Figure 1: Illustration of collection of open-loop and closed-loop trajectories. The gray part denotes the nominal optimal trajectory under ideal environments. For the collection of the open-loop trajectory (top), it is implicitly assumed that the noise process (denoted by the dashed arrow) only affects the measurement afterward. On the contrary, for the collection of closed-loop trajectory, the next control input will take into consideration this noise and make a correction (denoted by the red arrow).
• Figure 2: The difference between the gradients computed by PDP-based and proposed DDP-based algorithms on unconstrained problems.
• Figure 3: The computational time for each call of PDP-based and proposed DDP-based algorithms on unconstrained problems.
• Figure 4: The difference between the gradients computed by PDP-based and proposed DDP-based algorithms on constrained problems.
• Figure 5: The computational time for each call of PDP-based and proposed DDP-based algorithms on constrained problems.

Theorems & Definitions (18)

• Remark 3.1
• Remark 3.2
• Theorem 3.3
• proof
• Remark 3.4
• Theorem 3.5
• proof
• Remark 3.6
• Corollary 3.7
• Remark 3.8