Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Information Theoretic Active Inverse Reinforcement Learning

Ondrej Bajgar, Sid William Gould, Rohan Narayan Langford Mitta, Jonathon Liu, Oliver Newcombe, Jack Golden

TL;DR

The paper tackles learning human reward functions in autonomous systems by formulating active inverse reinforcement learning (IRL) that uses full-trajectory demonstrations. It adopts an information-theoretic objective, maximizing the expected information gain $\text{EIG}_n(\xi)$ to select the most informative environments, and uses nested Monte Carlo estimators alongside Bayesian optimization to efficiently allocate sampling budget. The approach is validated in two gridworld setups, showing that trajectory-based queries and Bayesian optimization substantially improve posterior reward inference and the resulting apprentice policy, compared to several baselines. This work provides a principled, scalable step toward active IRL in more general and continuous domains, with potential impact on safer and more reliable autonomous decision-making.

Abstract

As AI systems become increasingly autonomous, aligning their decision-making to human preferences is essential. In domains like autonomous driving or robotics, it is impossible to write down the reward function representing these preferences by hand. Inverse reinforcement learning (IRL) offers a promising approach to infer the unknown reward from demonstrations. However, obtaining human demonstrations can be costly. Active IRL addresses this challenge by strategically selecting the most informative scenarios for human demonstration, reducing the amount of required human effort. Where most prior work allowed querying the human for an action at one state at a time, we motivate and analyse scenarios where we collect longer trajectories. We provide an information-theoretic acquisition function, propose an efficient approximation scheme, and illustrate its performance through a set of gridworld experiments as groundwork for future work expanding to more general settings.

Toward Information Theoretic Active Inverse Reinforcement Learning

TL;DR

The paper tackles learning human reward functions in autonomous systems by formulating active inverse reinforcement learning (IRL) that uses full-trajectory demonstrations. It adopts an information-theoretic objective, maximizing the expected information gain to select the most informative environments, and uses nested Monte Carlo estimators alongside Bayesian optimization to efficiently allocate sampling budget. The approach is validated in two gridworld setups, showing that trajectory-based queries and Bayesian optimization substantially improve posterior reward inference and the resulting apprentice policy, compared to several baselines. This work provides a principled, scalable step toward active IRL in more general and continuous domains, with potential impact on safer and more reliable autonomous decision-making.

Abstract

As AI systems become increasingly autonomous, aligning their decision-making to human preferences is essential. In domains like autonomous driving or robotics, it is impossible to write down the reward function representing these preferences by hand. Inverse reinforcement learning (IRL) offers a promising approach to infer the unknown reward from demonstrations. However, obtaining human demonstrations can be costly. Active IRL addresses this challenge by strategically selecting the most informative scenarios for human demonstration, reducing the amount of required human effort. Where most prior work allowed querying the human for an action at one state at a time, we motivate and analyse scenarios where we collect longer trajectories. We provide an information-theoretic acquisition function, propose an efficient approximation scheme, and illustrate its performance through a set of gridworld experiments as groundwork for future work expanding to more general settings.
Paper Structure (18 sections, 10 equations, 3 figures)

This paper contains 18 sections, 10 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Task formulation
  3. Method
  4. Efficient sampling with Bayesian optimization
  5. Experiments
  6. Structured gridworld
  7. Fully random gridworld
  8. Results
  9. Discussion and conclusion
  10. Related work
  11. Scaling Properties
  12. Experiment details
  13. Bayesian IRL methods
  14. Baselines
  15. Experimental setup
  16. ...and 3 more sections

Figures (3)

  • Figure 1: (a) shows an illustrative gridworld and its true rewards. The lower left corner has a "jail" state with negative reward from which an agent cannot leave. The starred green state is the terminal "goal" state with a large positive reward. The brown, blue, and red states are "mud", "water", and "lava" type states respectively, whose rewards are unknown to the IRL agent. The IRL agent tries to learn the rewards of these three state types from expert demonstrations. (b) shows the learned distributions over the rewards of the "mud", "water", and "lava" state types respectively, at some particular step of the active learning process. These learned reward distributions are used to calculate the EIG of obtaining another expert demonstration starting from each given state, shown in (c). In this case, a demonstration starting in the bottom right state gives the most information about the unknown reward parameters.
  • Figure 2: Results on two gridworld environments comparing EIG-based methods with baselines. NMC stands for naive nested Monte Carlo estimation, while BO stands for Bayesian optimization. "Single st. EIG (x / 8.8)" denotes single-state EIG with the x axis scaled by 8.8 - the mean length of trajectories collected by the full-trajectory EIG variants.
  • Figure 3: These plots show that the EIG calculation step scales approximately quadratically in $n$, or linearly in the number of steps, and is very consistent. For comparison, the plot also shows the scaling of the time required to run the PolicyWalk algorithm for Bayesian IRL