Cost Function Estimation Using Inverse Reinforcement Learning with Minimal Observations

TL;DR

The paper tackles the ill-posed nature of IRL in continuous spaces by proposing MO-IRL, an iterative maximum-entropy IRL method that uses an OC solver to generate informative trajectories and per-trajectory weighting to approximate the partition function with minimal data. The approach computes incremental weight updates Delta w, accepts steps via an adaptive line search using merit criteria, and enhances learning through sub-sampling, elastic net regularization, and a moving window. Empirical results in point mass and robot manipulation tasks show MO-IRL achieves faster convergence and more accurate cost estimation than state-of-the-art methods PI2-IRL and IS-IRL, often with substantially fewer samples. These properties suggest MO-IRL is well suited for online cost learning and MPC-based control in human-in-the-loop or dynamic environments, where data efficiency and computation are critical.

Abstract

We present an iterative inverse reinforcement learning algorithm to infer optimal cost functions in continuous spaces. Based on a popular maximum entropy criteria, our approach iteratively finds a weight improvement step and proposes a method to find an appropriate step size that ensures learned cost function features remain similar to the demonstrated trajectory features. In contrast to similar approaches, our algorithm can individually tune the effectiveness of each observation for the partition function and does not need a large sample set, enabling faster learning. We generate sample trajectories by solving an optimal control problem instead of random sampling, leading to more informative trajectories. The performance of our method is compared to two state of the art algorithms to demonstrate its benefits in several simulated environments.

Figures (7)

• Figure 1: The performance of the IRL algorithms on three point mass environments (one per column). Results are shown in pairs of rows for MO-IRL, PI2-IRL, and IS-IRL. Top row of each pair shows the sampled trajectory set for each algorithm. Bottom rows of pairs show the set of OC rollouts based on the learned cost function. The initial optimal trajectory is shown in green in the bottom row. In the bottom rows, rollouts with obstacle collision are shown in red.
• Figure 2: Performance of MO-IRL and IS-IRL throughout their iterations. First column indicates the deviation of the resulting trajectory from optimality, second column shows the cost differences of the optimal and the solved trajectory given the estimated weight set, and the third column shows the cost difference of the optimal and solved trajectory given the optimal weights at each iteration. Vertical dashed lines show the termination of associated algorithms. PI2-IRL's performance is shown by a horizontal red dashed line.
• Figure 3: Comparison of each improving step for modifying the IRL performance. (a) shows the algorithm without any step acceptance, regularization, and sub-sampling. (b) is the same as (a) with sub-sampling. (c) shows (b) with added step acceptance method. (d) is (b) but with regularization. (e) shows the full version with sub-sampling, regularization, and step acceptance method.
• Figure 4: Sampled trajectory sets for the IRL approaches. MO-IRL carries out a different way of sampling that is not necessarily local to the optimal trajectory. PI2 IRL uses only the initial noisy local rollouts. IS-IRL concludes the iterations with a trajectory set containing both initial local rollouts, and OC's solutions. The green path seen on the left figure is the optimal trajectory.
• Figure 5: Iterative performance comparison between MO-IRL and IS-IRL. PI2-IRL's performance is indicated by the horizontal dashed red line.