Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interactive Explanations for Reinforcement-Learning Agents

Yotam Amitai, Ofra Amir, Guy Avni

TL;DR

The paper tackles the challenge of making reinforcement-learning agents explainable through interactive dialogue rather than static summaries. It introduces ASQ-IT, a system that retrieves agent-behavior clips by translating user questions into a fragment of $LTL_f$ and searching an offline library via automata-based processing. The authors demonstrate usability with lay users and show improved debugging performance against a static baseline, highlighting increased engagement and hypothesis exploration. This approach enables structured, user-driven explanations of sequential agent behavior with potential for integration into broader explanatory toolkits in safety-critical domains.

Abstract

As reinforcement learning methods increasingly amass accomplishments, the need for comprehending their solutions becomes more crucial. Most explainable reinforcement learning (XRL) methods generate a static explanation depicting their developers' intuition of what should be explained and how. In contrast, literature from the social sciences proposes that meaningful explanations are structured as a dialog between the explainer and the explainee, suggesting a more active role for the user and her communication with the agent. In this paper, we present ASQ-IT -- an interactive explanation system that presents video clips of the agent acting in its environment based on queries given by the user that describe temporal properties of behaviors of interest. Our approach is based on formal methods: queries in ASQ-IT's user interface map to a fragment of Linear Temporal Logic over finite traces (LTLf), which we developed, and our algorithm for query processing is based on automata theory. User studies show that end-users can understand and formulate queries in ASQ-IT and that using ASQ-IT assists users in identifying faulty agent behaviors.

Interactive Explanations for Reinforcement-Learning Agents

TL;DR

The paper tackles the challenge of making reinforcement-learning agents explainable through interactive dialogue rather than static summaries. It introduces ASQ-IT, a system that retrieves agent-behavior clips by translating user questions into a fragment of and searching an offline library via automata-based processing. The authors demonstrate usability with lay users and show improved debugging performance against a static baseline, highlighting increased engagement and hypothesis exploration. This approach enables structured, user-driven explanations of sequential agent behavior with potential for integration into broader explanatory toolkits in safety-critical domains.

Abstract

As reinforcement learning methods increasingly amass accomplishments, the need for comprehending their solutions becomes more crucial. Most explainable reinforcement learning (XRL) methods generate a static explanation depicting their developers' intuition of what should be explained and how. In contrast, literature from the social sciences proposes that meaningful explanations are structured as a dialog between the explainer and the explainee, suggesting a more active role for the user and her communication with the agent. In this paper, we present ASQ-IT -- an interactive explanation system that presents video clips of the agent acting in its environment based on queries given by the user that describe temporal properties of behaviors of interest. Our approach is based on formal methods: queries in ASQ-IT's user interface map to a fragment of Linear Temporal Logic over finite traces (LTLf), which we developed, and our algorithm for query processing is based on automata theory. User studies show that end-users can understand and formulate queries in ASQ-IT and that using ASQ-IT assists users in identifying faulty agent behaviors.

Paper Structure

This paper contains 31 sections, 2 theorems, 2 equations, 9 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. ASQ-IT: Running examples
  3. Domains
  4. Highway Domain (Fig. \ref{['fig:domains']}-Right)
  5. Frogger Domain (Fig. \ref{['fig:domains']}-Left)
  6. ASQ-IT usage: Example implementations
  7. ASQ-IT Interface
  8. Related Work
  9. Explanations in sequential decision-making settings
  10. Interactive explanations
  11. Explanations based on formal methods
  12. The Design of ASQ-IT
  13. Offline: Obtaining a database of clips
  14. Problem statement (informal)
  15. Domain expert mapping
  16. ...and 16 more sections

Key Result

Theorem 1

LTLf Consider an LTLf formula $\varphi$ over a set of predicates $P$. There is a DFA ${\cal A}_\varphi$ over the alphabet $\Sigma = 2^P$ whose language is the set of traces that $\varphi$ recognizes. That is, for every trace $\eta \in \Sigma^*$ we have $\eta \in L({\cal A})$ iff $\eta \models \varph

Figures (9)

  • Figure 1: Left: Frogger domain - agent is a green frog that seeks to reach the lilypads at the top of the screen while avoiding cars on the road and jumping between logs on the river. Right: Highway domain - agent is a green rectangular vehicle driving on a multi-lane highway while interacting with other vehicles and avoiding collision.
  • Figure 2: Example ASQ-IT implementation for the Highway domain. Left: Query interface; Right: Output generated videos. Query interpretation: "Agent starts at lane 1 behind some car, and finishes at lane 4 while somewhere along the way, the agent is no longer behind a car."
  • Figure 3: Example ASQ-IT implementation for the Frogger domain. Query interpretation: "Agent starts on the road with a car on its left and terminates at the river by drowning. Somewhere along its path a DOWN action takes place"
  • Figure 4: ASQ-IT Process Flow Diagram. Offline - Agent interaction traces are collected from simulations. An abstraction to domain-expert-defined predicates is applied to each state the agent encounters. Abstracted traces are saved to an Interaction Library; Online - User generates query through ASQ-IT's Quert Interface. Query is translated to formal expression and used to search Interaction Library for satisfying sub-traces. Video clips are generated from obtained sub-traces and presented to user.
  • Figure 5: Highway domain state abstraction. Left - An example frame from the simulation. The agent (green rectangle) is in the fourth lane (from the top) and behind another vehicle (blue rectangle); Right - The abstraction of the state to the domain-specific predicates.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Example 1
  • Example 2
  • Remark 1
  • Example 3
  • Example 4
  • Theorem 1
  • Remark 2
  • Theorem 2
  • Remark 3