Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BXRL: Behavior-Explainable Reinforcement Learning

Ram Rachum, Yotam Amitai, Yonatan Nakar, Reuth Mirsky, Cameron Allen

Abstract

A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such as "explain this specific action", "explain this specific trajectory", and "explain the entire policy". However, XRL lacks a formal definition for behavior as a pattern of actions across many episodes. We provide such a definition, and use it to enable a new query: "Explain this behavior". We present Behavior-Explainable Reinforcement Learning (BXRL), a new problem formulation that treats behaviors as first-class objects. BXRL defines a behavior measure as any function $m : Π\to \mathbb{R}$, allowing users to precisely express the pattern of actions that they find interesting and measure how strongly the policy exhibits it. We define contrastive behaviors that reduce the question "why does the agent prefer $a$ to $a'$?" to "why is $m(π)$ high?" which can be explored with differentiation. We do not implement an explainability method; we instead analyze three existing methods and propose how they could be adapted to explain behavior. We present a port of the HighwayEnv driving environment to JAX, which provides an interface for defining, measuring, and differentiating behaviors with respect to the model parameters.

BXRL: Behavior-Explainable Reinforcement Learning

Abstract

A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such as "explain this specific action", "explain this specific trajectory", and "explain the entire policy". However, XRL lacks a formal definition for behavior as a pattern of actions across many episodes. We provide such a definition, and use it to enable a new query: "Explain this behavior". We present Behavior-Explainable Reinforcement Learning (BXRL), a new problem formulation that treats behaviors as first-class objects. BXRL defines a behavior measure as any function , allowing users to precisely express the pattern of actions that they find interesting and measure how strongly the policy exhibits it. We define contrastive behaviors that reduce the question "why does the agent prefer to ?" to "why is high?" which can be explored with differentiation. We do not implement an explainability method; we instead analyze three existing methods and propose how they could be adapted to explain behavior. We present a port of the HighwayEnv driving environment to JAX, which provides an interface for defining, measuring, and differentiating behaviors with respect to the model parameters.
Paper Structure (38 sections, 7 equations, 3 figures, 4 tables)

This paper contains 38 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Explanation target and source
  4. Contrastive explanation
  5. Behavior as disposition
  6. Markov Decision Process (MDP)
  7. Attribution and importance methods
  8. Properties of Behavior
  9. Related Work
  10. rishav2025behavior
  11. HIGHLIGHTS amir2018highlights
  12. ASQ-IT amitai2024asqit
  13. Behavior-Explainable Reinforcement Learning (BXRL)
  14. Expressing a Behavior as a Number
  15. Adapting Existing XRL Methods for BXRL
  16. ...and 23 more sections

Figures (3)

  • Figure 1: HighJax TUI for defining behavior scenarios by selecting states and actions from existing rollouts.
  • Figure 2: Left: Return breakdown of a PPO agent training on HighJax. Training is deliberately slowed down to $D_{\mathrm{KL}} = 5 \times 10^{-4}$ per epoch. Right: Value of the collision behavior measure $m_c$ across training.
  • Figure 3: Six scenarios defining the collision behavior measure $m_c$. Each causes the ego car (dark gray) to crash within two timesteps. Columns correspond to the action taken: left, right, or faster.

Theorems & Definitions (2)

  • Definition 1: Behavior measure
  • Definition 2: BXRL Method