Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations

Myunsoo Kim, Hayeong Lee, Seong-Woong Shim, JunHo Seo, Byung-Jun Lee

TL;DR

NBDI addresses the limitation of fixed-length skills in long-horizon RL by introducing a state-action novelty module that identifies critical decision points for terminating skills. The method combines state novelty $\\chi(s)$ with conditional action novelty $\\chi(a|s)$ into a joint novelty $\\chi(s,a)$, estimated via ICM, enabling variable-length skills learned from task-agnostic demonstrations and deployed in downstream tasks through an SMDP framework. Key contributions include proposing a novelty-based termination criterion, showing robustness to substantial environment configuration changes, and demonstrating improved learning and transfer in navigation and manipulation tasks, as well as integration with skill-based meta-RL like SiMPL. The results highlight practical impact for sample-efficient policy learning and cross-task generalization, with potential extensions to more offline skill discovery scenarios and broader hierarchical RL settings.

Abstract

Intelligent agents are able to make decisions based on different levels of granularity and duration. Recent advances in skill learning enabled the agent to solve complex, long-horizon tasks by effectively guiding the agent in choosing appropriate skills. However, the practice of using fixed-length skills can easily result in skipping valuable decision points, which ultimately limits the potential for further exploration and faster policy learning. In this work, we propose to learn a simple and effective termination condition that identifies decision points through a state-action novelty module that leverages agent experience data. Our approach, Novelty-based Decision Point Identification (NBDI), outperforms previous baselines in complex, long-horizon tasks, and remains effective even in the presence of significant variations in the environment configurations of downstream tasks, highlighting the importance of decision point identification in skill learning.

NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations

TL;DR

NBDI addresses the limitation of fixed-length skills in long-horizon RL by introducing a state-action novelty module that identifies critical decision points for terminating skills. The method combines state novelty with conditional action novelty into a joint novelty , estimated via ICM, enabling variable-length skills learned from task-agnostic demonstrations and deployed in downstream tasks through an SMDP framework. Key contributions include proposing a novelty-based termination criterion, showing robustness to substantial environment configuration changes, and demonstrating improved learning and transfer in navigation and manipulation tasks, as well as integration with skill-based meta-RL like SiMPL. The results highlight practical impact for sample-efficient policy learning and cross-task generalization, with potential extensions to more offline skill discovery scenarios and broader hierarchical RL settings.

Abstract

Intelligent agents are able to make decisions based on different levels of granularity and duration. Recent advances in skill learning enabled the agent to solve complex, long-horizon tasks by effectively guiding the agent in choosing appropriate skills. However, the practice of using fixed-length skills can easily result in skipping valuable decision points, which ultimately limits the potential for further exploration and faster policy learning. In this work, we propose to learn a simple and effective termination condition that identifies decision points through a state-action novelty module that leverages agent experience data. Our approach, Novelty-based Decision Point Identification (NBDI), outperforms previous baselines in complex, long-horizon tasks, and remains effective even in the presence of significant variations in the environment configurations of downstream tasks, highlighting the importance of decision point identification in skill learning.
Paper Structure (49 sections, 1 theorem, 7 equations, 18 figures, 6 tables, 2 algorithms)

This paper contains 49 sections, 1 theorem, 7 equations, 18 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Option Framework
  4. Skill-based deep RL
  5. Novelty-based RL
  6. Background
  7. Markov Decision Process (MDP)
  8. Option Framework
  9. Simple and Effective Identification of Decision Points
  10. State-action Novelty-based Decision Point Identification
  11. Termination Improvement from State-action Novelty-based Terminations
  12. Learning Termination Conditions through State-action Novelty Module
  13. Problem Formulation
  14. Unsupervised Learning of State-action Novelty Module
  15. Reinforcement Learning with Skill Termination Conditions
  16. ...and 34 more sections

Key Result

Theorem 4.1

[Termination Improvement, sutton1998between, informal] For any meta-control policy $\mu$ on set of options $\mathcal{O}$, define a new set of options $\mathcal{O'}$, which is a set of options that we can additionally choose to terminate whenever the value of a state $V^\mu(s)$ is larger than the val

Figures (18)

  • Figure 1: Visualization of an example of critical decision points in the kitchen environment. High state-action novelty can be found in states where a subtask has been completed, and multiple subsequent subtasks are accessible. If termination occurs at a high state-action novelty point, the agent retains multiple plausible options. For example, after moving the kettle, it may choose to open the sliding cabinet or turn the oven knob (Left). Similarly, after flipping the light switch, the agent may proceed to open either the sliding cabinet or the left-hinged cabinet (Right).
  • Figure 2: Visualization of prediction error of ICM in maze and block stacking environment. Note the same offline data that is used to train ICM was used to compute this prediction error. (A), (B) and (C) are the state-action pairs with the highest prediction error, while (D), (E) and (F) are the ones with the lowest. Critical decision points---such as crossroads or states involving block manipulation---are typically associated with high prediction error. In contrast, low prediction errors are observed in states that are less important for decision-making.
  • Figure 3: The relative frequency of termination improvement occurrences (left), conditional action novelty (middle), and state novelty (right) in a small grid with three different goals. Higher percentile colors indicate a relatively greater number of termination improvement occurrences, higher conditional action novelty, and higher state novelty. Further details on the visualization procedure are provided in Appendix \ref{['implementation_details']}.
  • Figure 4: Performances of our method and baselines in solving downstream tasks. The shaded region represents 95% confidence interval across five different seeds. The last column of the table below illustrates the percentage improvement of our method over SPiRL.
  • Figure 5: Performances of our method and baselines in solving downstream tasks. The shaded region and error bar represents 95% confidence interval across five different seeds.
  • ...and 13 more figures

Theorems & Definitions (1)

  • Theorem 4.1