A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps

TL;DR

This work uses DINO, a self-supervised Vision Transformer that produces attention maps strikingly similar to human gaze patterns, to explore a saccade inspired method to focus the processing of information on key regions in visual space, and demonstrates that DINO provides superior fixation guidance for selecting informative regions.

Abstract

Human vision achieves remarkable perceptual performance while operating under strict metabolic constraints. A key ingredient is the selective attention mechanism, driven by rapid saccadic eye movements that constantly reposition the high-resolution fovea onto task-relevant locations, unlike conventional AI systems that process entire images with equal emphasis. Our work aims to draw inspiration from the human visual system to create smarter, more efficient image processing models. Using DINO, a self-supervised Vision Transformer that produces attention maps strikingly similar to human gaze patterns, we explore a saccade inspired method to focus the processing of information on key regions in visual space. To do so, we use the ImageNet dataset in a standard classification task and measure how each successive saccade affects the model's class scores. This selective-processing strategy preserves most of the full-image classification performance and can even outperform it in certain cases. By benchmarking against established saliency models built for human gaze prediction, we demonstrate that DINO provides superior fixation guidance for selecting informative regions. These findings highlight Vision Transformer attention as a promising basis for biologically inspired active vision and open new directions for efficient, neuromorphic visual processing.

Figures (17)

• Figure 1: Schematic illustration of attention map extraction within a single transformer layer. (a.) The input set of token embeddings, including the additional [CLS] token, is passed trough the multiple attention heads for parallel processing. (b.) Within each head, the tokens are projected through learned linear layers to obtain Query, Key, and Value representations (Value projections omitted in the figure for clarity). (c.) Attention weights are computed as the scaled dot product between Queries and Keys (as depicted in the bottom pink frame). The last column of the resulting attention matrix corresponds to the [CLS] token, and yields a vector of attention scores indicating the relative importance assigned to each image patch. (d.) This vector is reshaped into a 2-D grid to recover the spatial layout of the original image. This operation outputs one spatial attention map per head.
• Figure 2: Saccade selection method: (a.) The input image of dimension $H \times W$ is split into $\dfrac{H}{16} \times \dfrac{W}{n}$ sized patches and embedded into token vectors. (b.) The tokens are passed through the DINO transformer, and attention flow from patch tokens to [CLS] token (white arrows) are extracted and reshaped into one attention map per attention-head. (c.) The multiple attention maps are fused into one by taking the maximum value across heads. (d.) The highest-attention locations define square regions ("saccades") whose tokens are retained. (e.) Selected regions are revealed sequentially, and the image variants are classified by a pre-trained linear head. Image adapted from Wikimedia Commons gdml_saïmiri
• Figure 3: Accuracy score across saccades: For a fovea of size 3x3 tokens (a.), 5x5 tokens (b.), and the corresponding percentage of the image given to the model throughout saccades (c.). The accuracy score when the model is given the full image is shown as a dotted red line. The light curves correspond to the random saccades in every plot.
• Figure 4: Histogram of classification dynamics across saccades: a. and b. Cumulative accuracy (percentage of images correctly classified at least once), with the proportion of images that turned correct or incorrect at each saccade. c. and d.. Percentages of images that changed classification at this specific saccade (stacked bars) (Correct$\rightarrow{}$Wrong and Wrong$\rightarrow{}$Correct) and the proportions that have remained incorrect (Stayed Wrong) or remained recovered to correct (Stayed Right). e. and f. Percentage of images distributed according to the number of occurrence of correct classification across saccades, over the full validation dataset, and over the subset of images that were correctly classified during saccades but not when the image was given full (g. and h.). The percentage correspond to the percentage over the whole validation dataset in every case. a., c., e. and g. : $3 \times 3$ tokens fovea. b., d., f. and h.: $5 \times{} 5$ tokens fovea
• Figure 5: Evolution of the model’s classification certainty across saccades: Curves represent the mean certainty for attention driven (a.) and random saccades (b.). Colors indicate sets of images correctly classified at a given saccade (dark to light blue: first to tenth saccade). The red dotted line shows the mean certainty of all images, across saccade. Colored bars in the rectangles on the right of each plot shows mean certainty when the full image is given to the model, for each set.