Revealing Human Attention Patterns from Gameplay Analysis for Reinforcement Learning

Abstract

This study introduces a novel method for revealing human internal attention patterns (decision-relevant attention) from gameplay data alone, leveraging offline attention techniques from reinforcement learning (RL). We propose contextualized, task-relevant (CTR) attention networks, which generate attention maps from both human and RL agent gameplay in Atari environments. To evaluate whether the human CTR maps reveal internal attention patterns, we validate our model by quantitative and qualitative comparison to the agent maps as well as to a temporally integrated overt attention (TIOA) model based on human eye-tracking data. Our results show that human CTR maps are more sparse than the agent ones and align better with the TIOA maps. Following a qualitative visual comparison we conclude that they likely capture patterns of internal attention. As a further application, we use these maps to guide RL agents, finding that human attention-guided agents achieve slightly improved and more stable learning compared to baselines, and significantly outperform TIOA-based agents. This work advances the understanding of human-agent attention differences and provides a new approach for extracting and validating internal attention patterns from behavioral data.

Figures (11)

• Figure 1: The goal of this study is to reveal human internal attention patterns (decision-relevant attention) solely from gameplay analysis: An attention model is proposed that extracts contextually task-relevant features from human and reinforcement learning (RL) agent gameplay. A temporally integrated gaze prediction model is used as a point of comparison to confirm that the attention model successfully reveals human internal attention patterns.
• Figure 2: a) Contextualized task-relevant (CTR) attention network as well as temporally-integrated overt attention (TIOA) network architecture; b) Autoencoder (AE) architecture; c) Action predictor architecture; d) Method of creating a blended feature space $\mathcal{F}_\mathrm{B}$ from a target and source for the action predictor; e) Exmple attention map and ablation study: Example frame and attention map (left), and log$_{10}$ histogram of activations (right) over different sparsity control factors $\lambda$ for the human CTR attention network with and without feature blending. Feature blending drives the activation distribution toward binarization (values near 0 or 1).
• Figure 3: Step-by-step construction of a temporally-integrated overt attention target map from gaze positions for training of the TIOA network.
• Figure 4: Action predictor accuracies over unseen attention-masked states for human (CTR and TIOA) and agent (only CTR) gameplay at different attention activation rates/ sparsities. Dashed lines indicate the minimum activation rate where 97% of the maximum action prediction accuracy is reached.
• Figure 5: Relative percentage decrease in action prediction accuracy from maximum comparing human CTR vs. agent CTR across different sparsity levels. Mean and 95% confidence interval across six games. Agent attention shows approximately twice the relative accuracy decrease compared to human attention, supporting that agents have more distributed attention.