Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Transparent Reward Models via Unsupervised Feature Selection

Daulet Baimukashev, Gokhan Alcan, Kevin Sebastian Luck, Ville Kyrki

TL;DR

This work proposes a novel approach to construct compact and transparent reward models from automatically selected state features that enable the learning of policies that closely match expert behavior by training standard reinforcement learning algorithms from scratch.

Abstract

In complex real-world tasks such as robotic manipulation and autonomous driving, collecting expert demonstrations is often more straightforward than specifying precise learning objectives and task descriptions. Learning from expert data can be achieved through behavioral cloning or by learning a reward function, i.e., inverse reinforcement learning. The latter allows for training with additional data outside the training distribution, guided by the inferred reward function. We propose a novel approach to construct compact and transparent reward models from automatically selected state features. These inferred rewards have an explicit form and enable the learning of policies that closely match expert behavior by training standard reinforcement learning algorithms from scratch. We validate our method's performance in various robotic environments with continuous and high-dimensional state spaces. Webpage: \url{https://sites.google.com/view/transparent-reward}.

Learning Transparent Reward Models via Unsupervised Feature Selection

TL;DR

This work proposes a novel approach to construct compact and transparent reward models from automatically selected state features that enable the learning of policies that closely match expert behavior by training standard reinforcement learning algorithms from scratch.

Abstract

In complex real-world tasks such as robotic manipulation and autonomous driving, collecting expert demonstrations is often more straightforward than specifying precise learning objectives and task descriptions. Learning from expert data can be achieved through behavioral cloning or by learning a reward function, i.e., inverse reinforcement learning. The latter allows for training with additional data outside the training distribution, guided by the inferred reward function. We propose a novel approach to construct compact and transparent reward models from automatically selected state features. These inferred rewards have an explicit form and enable the learning of policies that closely match expert behavior by training standard reinforcement learning algorithms from scratch. We validate our method's performance in various robotic environments with continuous and high-dimensional state spaces. Webpage: \url{https://sites.google.com/view/transparent-reward}.

Paper Structure

This paper contains 24 sections, 21 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Feature construction for IRL
  4. Adversarial IRL
  5. Problem Statement
  6. Proposed Method
  7. Trajectory probability under maximum entropy
  8. Pseudo labels
  9. Candidate features
  10. Feature selection
  11. Reward and policy retrieval
  12. Experiments
  13. Setup
  14. Data collection
  15. Baselines
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Current IRL methods require hand-picked features to use as reward components. Adversarial methods learn the reward using neural networks which are less transparent and not amendable. But, the proposed method finds automatically relevant set of state features and constructs transparent reward models.
  • Figure 2: Benchmark tasks used in this paper.
  • Figure 3: IRL training curves for the Walker2d task comparing different feature selection baselines.
  • Figure 4: Comparison of training with ground truth and recovered reward functions for HalfCheetah.