Neuroevolution of Self-Attention Over Proto-Objects

TL;DR

The paper addresses the computational burden of attention in vision-based reinforcement learning by replacing patch-based attention with proto-objects derived from segmentation as attention tokens. It introduces a five-stage proto-object attentional agent (convolution, quantization, segmentation, attention, control) optimized with CMA-ES, selecting a single proto-object to feed an LSTM controller. Results on Car Racing and Doom Take Cover show competitive or superior performance with 62% fewer parameters and substantially faster training, highlighting the efficiency of semantically rich bottlenecks. The work demonstrates the feasibility of neuroevolutionary approaches to non-differentiable components and discusses future directions toward differentiable training, memory-decoupled controllers, and scalability to real-world images.

Abstract

Proto-objects - image regions that share common visual properties - offer a promising alternative to traditional attention mechanisms based on rectangular-shaped image patches in neural networks. Although previous work demonstrated that evolving a patch-based hard-attention module alongside a controller network could achieve state-of-the-art performance in visual reinforcement learning tasks, our approach leverages image segmentation to work with higher-level features. By operating on proto-objects rather than fixed patches, we significantly reduce the representational complexity: each image decomposes into fewer proto-objects than regular patches, and each proto-object can be efficiently encoded as a compact feature vector. This enables a substantially smaller self-attention module that processes richer semantic information. Our experiments demonstrate that this proto-object-based approach matches or exceeds the state-of-the-art performance of patch-based implementations with 62% less parameters and 2.6 times less training time.

Figures (8)

• Figure 1: Our attentional agent is capable of focusing on entire uniform regions instead of small fixed-size image patches.
• Figure 2: The agent extracts patches from the current frame and selects the top-$k$ ones (highlighted in white) by a hard attention mechanism. By visualizing the selected patches, it is possible to have a better idea of the learned strategy, as well as making it easier for debugging.
• Figure 3: Flowchart of our complete process. In our experiments, $H=W=96$, $F=11$ (number of segment features), $d_q=2$, $M=2$ (we use only the $x,y$ coordinates from each token), $k=1$ and $O=3$ (number of outputs for both environments). Any number of convolutional layers can be used, as long as they preserve image size and the last one has 3 filters to match the residual connection. We use one layer of 3 1x1 filters. Our quantization is set to 1 bit per channel (8 colors). $P$ is the $PReLU$ activation function. Green elements are new in relation to attentionagent2020.
• Figure 4: Learning curve comparison in the Car Racing environment over 400 test runs out of training sample tracks for each shown generation. Our proto-object method achieves significantly better results after 1000 generations in relation to the patch-based of attentionagent2020. Both peaked at 1000 generations.
• Figure 5: Evolution of segmentation at 5 relevant points in time. The basic attention strategy (focus on the smaller grass region, highlighted in white with centroid in black; other centroids in pink) is learned early in the process, while segmentation keeps evolving until the end. Top-Left: Raw image after resizing and before segmentation. Top-Center (100 generations): The trivial segmentation from original colors is kept for the first couple hundred generations. Top-Right (300 generations): It learns to separate the track from the head-up display (HUD) below and differentiates some of the ABS sensors. Bottom-Left (800 generations): It swaps the track and HUD segmentations, making the car visible, differentiating the red corner markers and hiding the gyroscope indicators. It also differentiates the corner white markings from the score and speed indicator, and merges the ABS sensors again. Bottom-Center (900 generations): It differentiates the track segments. Bottom-Right (1000 generations): It gives up on the fine-grained segmentation of the track and goes back to the previous segmentation strategy, with very small and irrelevant changes in some pixels in the score.