Learning Rules from Rewards

TL;DR

This work introduces the Relational Regression Tree Learner (RRTL), a model that incrementally learns ground relational policies for reinforcement learning by selecting task-relevant relations from a broad candidate set. It compares two split strategies—logical and comparative—using an F-test criterion and demonstrates that comparative splits yield more robust learning across three relationally challenging Atari games (Breakout, Pong, Demon Attack). The results show that RRTL can form simple, effective relational policies that leverage structured representations to guide adaptive behavior, with comparative-splits often offering greater consistency and higher performance. The study discusses limitations of the current frequentist split approach and outlines future avenues, including Bayesian splitting, improved action representations, and integration with schema induction and analogy to enhance relational learning and generalization.

Abstract

Humans can flexibly generalize knowledge across domains by leveraging structured relational representations. While prior research has shown how such representations support analogical reasoning, less is known about how they are recruited to guide adaptive behavior. We address this gap by introducing the Relational Regression Tree Learner (RRTL), a model that incrementally builds policies over structured relational inputs by selecting task-relevant relations during the learning process. RRTL is grounded in the framework of relational reinforcement learning but diverges from traditional approaches by focusing on ground (i.e., non-variabilized) rules that refer to specific object configurations. Across three Atari games of increasing relational complexity (Breakout, Pong, Demon Attack), the model learns to act effectively by identifying a small set of relevant relations from a broad pool of candidate relations. A comparative version of the model, which partitions the state space using relative magnitude values (e.g., "more", "same", "less"), showed more robust learning than a version using logical (binary) splits. These results provide a proof of principle that reinforcement signals can guide the selection of structured representations, offering a computational framework for understanding how relational knowledge is learned and deployed in adaptive behavior.

Figures (6)

• Figure 1: Two different ways of representing state-action values. Panel A shows three potential states of the Breakout environment where the player is either to the left of the ball, at the same x-coordinate, or to the right of the ball. The fact that the value of the action RIGHT is higher when the player is to the left of the ball, lower when the player and the ball are at the same x-coordinate, and lowest when the player is to the right of the ball, can be represented as a logical relational regression tree (Panel B) or as a comparative relational regression tree (Panel C). See text for details.
• Figure 2: A typical state of the Pong environment.
• Figure 3: Two typical states of the Demon Attack environment. Panel A shows a state where there are three big enemies that can shoot missiles. Panel B shows a state where, besides big enemies, there are small enemies. The player loses a life if it is touched by a small enemy.
• Figure 4: Training results by model version and game. The y-axis corresponds to the moving average of the human-normalized returns. For each game, the size of the moving window was set to the maximum number of training episodes divided by 12, and the minimum number of observations was set to the same number divided by 50. Each line corresponds to an individual random seed. Each line is colored by the returns' variance, with darker colors indicating higher variance.
• Figure 5: Test results by model version and game. The first row shows the median performance as a line plot, along with the individual random-seed data points. Error bars are 95% confidence intervals. The second row shows the corresponding frequency distributions.