Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions

Rashmeet Kaur Nayyar, Naman Shah, Siddharth Srivastava

TL;DR

Context-sensitive abstractions for RL with parameterized actions (SPA-CAT) enable online, hierarchical refinement of both state and action parameter spaces to improve learning efficiency in long-horizon, sparse-reward tasks. The approach combines Action Parameter Trees (APTs) with State and Parameterized Action Conditional Abstraction Trees, and learns refinements using a heterogeneous signal that blends TD-error dispersion and value dispersion via an annealing parameter $\beta$, all integrated with $TD(\lambda)$. PEARL alternates between a Learning phase that updates abstract Q-values and a Refinement phase that expands the abstraction where heterogeneity is high, including both uniform and flexible refinement strategies. Empirical results across OfficeWorld, Pinball, Multi-City Transport, and Soccer Goal show that PEARL-flexible achieves the best sample efficiency and policy performance among baselines like MP-DQN and HyAR, demonstrating the practical value of jointly learned state-action abstractions for parameterized actions. The work provides a principled framework for exploiting latent structure in parameterized-action spaces, reducing the need for environment-specific handcrafting and enabling scalable RL in complex domains.

Abstract

Real-world sequential decision-making often involves parameterized action spaces that require both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. Existing approaches exhibit severe limitations in this setting -- planning methods demand hand-crafted action models, and standard reinforcement learning (RL) algorithms are designed for either discrete or continuous actions but not both, and the few RL methods that handle parameterized actions typically rely on domain-specific engineering and fail to exploit the latent structure of these spaces. This paper extends the scope of RL algorithms to long-horizon, sparse-reward settings with parameterized actions by enabling agents to autonomously learn both state and action abstractions online. We introduce algorithms that progressively refine these abstractions during learning, increasing fine-grained detail in the critical regions of the state-action space where greater resolution improves performance. Across several continuous-state, parameterized-action domains, our abstraction-driven approach enables TD($λ$) to achieve markedly higher sample efficiency than state-of-the-art baselines.

Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions

TL;DR

Context-sensitive abstractions for RL with parameterized actions (SPA-CAT) enable online, hierarchical refinement of both state and action parameter spaces to improve learning efficiency in long-horizon, sparse-reward tasks. The approach combines Action Parameter Trees (APTs) with State and Parameterized Action Conditional Abstraction Trees, and learns refinements using a heterogeneous signal that blends TD-error dispersion and value dispersion via an annealing parameter , all integrated with . PEARL alternates between a Learning phase that updates abstract Q-values and a Refinement phase that expands the abstraction where heterogeneity is high, including both uniform and flexible refinement strategies. Empirical results across OfficeWorld, Pinball, Multi-City Transport, and Soccer Goal show that PEARL-flexible achieves the best sample efficiency and policy performance among baselines like MP-DQN and HyAR, demonstrating the practical value of jointly learned state-action abstractions for parameterized actions. The work provides a principled framework for exploiting latent structure in parameterized-action spaces, reducing the need for environment-specific handcrafting and enabling scalable RL in complex domains.

Abstract

Real-world sequential decision-making often involves parameterized action spaces that require both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. Existing approaches exhibit severe limitations in this setting -- planning methods demand hand-crafted action models, and standard reinforcement learning (RL) algorithms are designed for either discrete or continuous actions but not both, and the few RL methods that handle parameterized actions typically rely on domain-specific engineering and fail to exploit the latent structure of these spaces. This paper extends the scope of RL algorithms to long-horizon, sparse-reward settings with parameterized actions by enabling agents to autonomously learn both state and action abstractions online. We introduce algorithms that progressively refine these abstractions during learning, increasing fine-grained detail in the critical regions of the state-action space where greater resolution improves performance. Across several continuous-state, parameterized-action domains, our abstraction-driven approach enables TD() to achieve markedly higher sample efficiency than state-of-the-art baselines.
Paper Structure (31 sections, 6 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 31 sections, 6 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. TD($\lambda$)
  4. Abstraction
  5. Our Approach
  6. Running example
  7. State and Action Abstractions
  8. Learning Abstraction Trees
  9. Uniform refinement
  10. Learning flexible refinements
  11. PEARL Algorithm
  12. Learning phase
  13. Refinement phase
  14. Empirical Results
  15. Test environments
  16. ...and 16 more sections

Figures (7)

  • Figure 1: In a continuous version of the office domain, the agent needs to learn policies for delivering multiple items. Polygonal cells illustrate learned state abstractions, and arrows illustrate learned policies with abstract actions parameterized by parameter intervals. Each arrow corresponds to an interval $[a,b)$ of possible movement values: the solid segment indicates the lower bound $a$, and the dotted segment indicates the interval width $b-a$. Narrower dotted segments denote higher precision in the learned action parameters.
  • Figure 2: Illustration of a SPA-CAT for Office World.
  • Figure 3: Learned state abstractions using flexible (left) and uniform (right) refinement strategies. The agent is at the top left; it must deliver both coffee and mail to the bottom right. Black lines and regions indicate obstacles. Colors represent actions (yellow: right, green: down, red: up, blue: left).
  • Figure 4: (a) Office World: The robot needs to pickup coffee and mail and deliver to the office. (b) Pinball: A small, dynamic ball needs to be manouvered into a red hole, avoiding collisions with irregularly shaped obstacles. (c) Soccer Goal: The white agent needs to kick the small black ball past the red keeper. (d) Multi-City Transport: The agent needs to collect a package from a designated location (marked by blue) in a city and deliver to a target airport (marked by red) in a different city. Cities are connected only via airports.
  • Figure 5: Comparison of PEARL-flexible and PEARL-uniform with MP-DQN and HyAR in four domains: Office World, Pinball, Multi-City Transport, and Soccer Goal with mean and standard deviation across 50 independent trials.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Definition 3.1: Action Parameter Tree (APT)
  • Definition 3.2: State and Parameterized Action Conditional Abstraction Tree (SPA-CAT)