Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metrics

TL;DR

This work addresses the opacity of RL learning beyond reward signals by introducing ATOMs, attention-oriented metrics derived from $LRP$ saliency maps, to track how agents allocate attention during training. It defines two metrics, hierarchical-attention and combinatorial-attention, and validates them through three Pong variations and a Dual Ball Discrimination Test to link attention with behaviour. Results show game-specific attention patterns that align with observed actions and reveal a phased learning trajectory where paddle-focused attention emerges alongside performance gains. ATOMs offer a quantitative, controllable framework to understand attention-learning dynamics and could aid in diagnosing observational overfitting and guiding exploration in RL systems, with broader implications for neuroscience-inspired perspectives on learning.

Abstract

The learning process of a reinforcement learning (RL) agent remains poorly understood beyond the mathematical formulation of its learning algorithm. To address this gap, we introduce attention-oriented metrics (ATOMs) to investigate the development of an RL agent's attention during training. In a controlled experiment, we tested ATOMs on three variations of a Pong game, each designed to teach the agent distinct behaviours, complemented by a behavioural assessment. ATOMs successfully delineate the attention patterns of an agent trained on each game variation, and that these differences in attention patterns translate into differences in the agent's behaviour. Through continuous monitoring of ATOMs during training, we observed that the agent's attention developed in phases, and that these phases were consistent across game variations. Overall, we believe that ATOM could help improve our understanding of the learning processes of RL agents and better understand the relationship between attention and learning.

Figures (10)

• Figure 1: a ATOMs pipeline. b Variations of the Pong game. Red balls yield reward to the agent, the gray ball is a distractor. The balls' dynamics are illustrated with lines indicating if a ball bounces back from the opponent or passes through it. c Dual Ball Discrimination Test, which forces the agent to choose between B1 or B2.
• Figure 2: a Hierarchical-metric averaged over all agents. b Dissimilarity matrix for the hierarchy-attention metrics for all agents.c Combinatorial-attention metric which examines co-observation of objects. Each object is symbolized by a distinct coloured dot. Combinations of objects are indicated through the simultaneous colouring of mutliple dots. d Dissimilarity matrix for the combinatorial-attention metrics for all agents. e Relative interaction of the agent with B1 with respect to B2 in function of the relative hierarchy of B1 with respect to B2. v0 was added as a reference.
• Figure 3: Performance score and ATOMs during learning. The agent's performance score was averaged (dark line) over ten games at each measurement (the standard deviation is represented with the shaded region). Dotted lines mark periods of notable score improvement, selected manually. The combinatorial-attention grouped by category: noise (grey), combinations of balls only (orange), combinations of opponent and balls (cyan),and combinations of objects including the agent (red).
• Figure 4: Illustration of the extraction of the relevance score for the ball object computed from a neuron $k$ in the $Fc$ layer. Here a single frame (among the 4 frames constituting an input) is represented for clarity purposes. The relevance score of neuron $k$ with respect to the output of the network is computed with a first LRP operation noted here as $LRP_1$. The relevance score of the ball object with respect to the neuron $k$ is then computed with a second LRP operation noted here as $LRP_2$.
• Figure 5: Illustrations of different combinatorial-attention results. Each line represents a combination of objects "looked at" by a neuron in the $F_c$ layer. From left to right: symbol of the combination as depicted in the main text, neuron's relevance map overlaid onto the input frames, bar plot quantifying the average intensity of relevance scores for each object with the 25% threshold.