Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-Allied Relational Reinforcement Learning

Fateme Golivand Darvishvand, Hikaru Shindo, Sahil Sidheekh, Kristian Kersting, Sriraam Natarajan

TL;DR

The paper tackles the challenge of learning symbolic, relational policies in both structured and unstructured domains, where traditional deep RL struggles with generalization and interpretability. It proposes RAEL, a framework that combines Relational RL with object-centric representations and active human advice, guided by a budget on expert queries and a soft constraint incorporation via policy reuse. RAEL uses Relational Fitted Q-learning (RFQ) with gradient-boosted relational regression trees and an object-centric extractor to convert unstructured inputs into symbolic states, while actively querying a human in high-uncertainty states to improve sample efficiency and policy fidelity. Experiments across Atari-based sub-symbolic tasks and the Blocks World domain demonstrate improved performance, faster convergence with abstraction-guided advice, and the practical viability of human-in-the-loop, symbolic RL for complex environments.

Abstract

Reinforcement learning (RL) has experienced a second wind in the past decade. While incredibly successful in images and videos, these systems still operate within the realm of propositional tasks ignoring the inherent structure that exists in the problem. Consequently, relational extensions (RRL) have been developed for such structured problems that allow for effective generalization to arbitrary number of objects. However, they inherently make strong assumptions about the problem structure. We introduce a novel framework that combines RRL with object-centric representation to handle both structured and unstructured data. We enhance learning by allowing the system to actively query the human expert for guidance by explicitly modeling the uncertainty over the policy. Our empirical evaluation demonstrates the effectiveness and efficiency of our proposed approach.

Human-Allied Relational Reinforcement Learning

TL;DR

The paper tackles the challenge of learning symbolic, relational policies in both structured and unstructured domains, where traditional deep RL struggles with generalization and interpretability. It proposes RAEL, a framework that combines Relational RL with object-centric representations and active human advice, guided by a budget on expert queries and a soft constraint incorporation via policy reuse. RAEL uses Relational Fitted Q-learning (RFQ) with gradient-boosted relational regression trees and an object-centric extractor to convert unstructured inputs into symbolic states, while actively querying a human in high-uncertainty states to improve sample efficiency and policy fidelity. Experiments across Atari-based sub-symbolic tasks and the Blocks World domain demonstrate improved performance, faster convergence with abstraction-guided advice, and the practical viability of human-in-the-loop, symbolic RL for complex environments.

Abstract

Reinforcement learning (RL) has experienced a second wind in the past decade. While incredibly successful in images and videos, these systems still operate within the realm of propositional tasks ignoring the inherent structure that exists in the problem. Consequently, relational extensions (RRL) have been developed for such structured problems that allow for effective generalization to arbitrary number of objects. However, they inherently make strong assumptions about the problem structure. We introduce a novel framework that combines RRL with object-centric representation to handle both structured and unstructured data. We enhance learning by allowing the system to actively query the human expert for guidance by explicitly modeling the uncertainty over the policy. Our empirical evaluation demonstrates the effectiveness and efficiency of our proposed approach.
Paper Structure (12 sections, 5 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 12 sections, 5 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Reinforcement Learning (RL)
  4. Relational RL (RRL).
  5. Object Centric Representations.
  6. Human Advice in RL.
  7. Relational Active Advice Elicitation in Reinforcement Learning (RAEL)
  8. Experiments and Results
  9. Related Work
  10. Relational and Neuro-Symbolic RL.
  11. Human Allied RL and Advice Integration.
  12. Conclusion

Figures (3)

  • Figure 1: Overview of the proposed Relational Active Advice Elicitation (RAEL) framework. The environment provides either raw or symbolic states; an object-centric extraction module lifts raw images to symbolic representations. The agent’s behavior is determined by a probabilistic policy reuse strategy, combining relational fitted Q-learning with $\epsilon$-greedy exploration and active advice from an expert. After each training cycle, policy rollouts are used to compute statewise uncertainty. The most uncertain states are presented to the expert for advice, which is stored as abstractions for reuse. Advice-aware Q-value updates incorporate this guidance to improve learning efficiency.
  • Figure 2: Active advice improves performance in Blocks World. Cumulative reward as a function of training iteration for the stacking task. The orange curve shows mean performance with active advice (entropy-based querying), while the blue curve shows training without advice. Active advice consistently yields higher rewards and more stable learning compared to the advice-free baseline.
  • Figure 3: Impact of abstraction-guided advice in Atari. Cumulative reward over training iterations for (a) Kangaroo and (b) Seaquest. The blue curve shows RAEL with active advice including relational abstraction, the orange curve is active advice with only action preferences, and the green curve is no advice. Across both domains, advice with abstraction yields faster and higher returns, demonstrating improved sample efficiency and policy quality.

Theorems & Definitions (1)

  • Definition 1