Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autonomous state-space segmentation for Deep-RL sparse reward scenarios

Gianluca Maselli, Vieri Giuliano Santucci

TL;DR

The paper tackles reinforcement learning in sparse reward settings typical of open-ended autonomous systems. It introduces a two-level architecture that combines an Intrinsic Curiosity Module for exploration with autonomous sub-goal discovery and sub-goal policy learning (PTR). The authors validate the approach in the Gym SuperMarioBros environment with dense rewards disabled, showing efficient exploration and robust policy learning without task-specific priors. Key contributions include competence-guided sub-goal selection, automatic state-space segmentation into meaningful sub-goals, and improved final-goal achievement compared with purely intrinsic exploration, with implications for open-ended learning and robotics.

Abstract

Dealing with environments with sparse rewards has always been crucial for systems developed to operate in autonomous open-ended learning settings. Intrinsic Motivations could be an effective way to help Deep Reinforcement Learning algorithms learn in such scenarios. In fact, intrinsic reward signals, such as novelty or curiosity, are generally adopted to improve exploration when extrinsic rewards are delayed or absent. Building on previous works, we tackle the problem of learning policies in the presence of sparse rewards by proposing a two-level architecture that alternates an ''intrinsically driven'' phase of exploration and autonomous sub-goal generation, to a phase of sparse reward, goal-directed policy learning. The idea is to build several small networks, each one specialized on a particular sub-path, and use them as starting points for future exploration without the need to further explore from scratch previously learnt paths. Two versions of the system have been trained and tested in the Gym SuperMarioBros environment without considering any additional extrinsic reward. The results show the validity of our approach and the importance of autonomously segment the environment to generate an efficient path towards the final goal.

Autonomous state-space segmentation for Deep-RL sparse reward scenarios

TL;DR

The paper tackles reinforcement learning in sparse reward settings typical of open-ended autonomous systems. It introduces a two-level architecture that combines an Intrinsic Curiosity Module for exploration with autonomous sub-goal discovery and sub-goal policy learning (PTR). The authors validate the approach in the Gym SuperMarioBros environment with dense rewards disabled, showing efficient exploration and robust policy learning without task-specific priors. Key contributions include competence-guided sub-goal selection, automatic state-space segmentation into meaningful sub-goals, and improved final-goal achievement compared with purely intrinsic exploration, with implications for open-ended learning and robotics.

Abstract

Dealing with environments with sparse rewards has always been crucial for systems developed to operate in autonomous open-ended learning settings. Intrinsic Motivations could be an effective way to help Deep Reinforcement Learning algorithms learn in such scenarios. In fact, intrinsic reward signals, such as novelty or curiosity, are generally adopted to improve exploration when extrinsic rewards are delayed or absent. Building on previous works, we tackle the problem of learning policies in the presence of sparse rewards by proposing a two-level architecture that alternates an ''intrinsically driven'' phase of exploration and autonomous sub-goal generation, to a phase of sparse reward, goal-directed policy learning. The idea is to build several small networks, each one specialized on a particular sub-path, and use them as starting points for future exploration without the need to further explore from scratch previously learnt paths. Two versions of the system have been trained and tested in the Gym SuperMarioBros environment without considering any additional extrinsic reward. The results show the validity of our approach and the importance of autonomously segment the environment to generate an efficient path towards the final goal.

Paper Structure

This paper contains 4 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Implemented solution and experimental scenario
  3. Compared systems
  4. Results and Discussion

Figures (4)

  • Figure 1: The experimental scenario
  • Figure 2: The proposed architecture.
  • Figure 3: The paths generated by the compared systems of one representative seed
  • Figure 4: Competence curves (over number of training episodes) over the discovered sub-goals in the two compared systems. Data refers to the same representative seeds used in Fig.3