Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emergence of Fixational and Saccadic Movements in a Multi-Level Recurrent Attention Model for Vision

Pengcheng Pan, Yonekura Shogo, Yasuo Kuniyoshi

TL;DR

The paper tackles the mismatch between current hard attention models and human visual exploration by introducing MRAM, a two-layer recurrent attention framework that decouples gaze control from final classification. The lower layer governs fast glimpse updates (saccade-like decisions) while the upper layer handles slow, high-level recognition, with a hybrid baseline stabilizing training through feedback-like signals. Experiments on MNIST, FashionMNIST, and FER2013 show MRAM not only matches or exceeds baseline accuracy but also yields emergent eye-movement patterns resembling human fixations and saccades, aligning with empirical eye-tracking data. The results suggest that hierarchical, biologically inspired attention can improve both interpretability and task performance, while highlighting areas for enhancement in peripheral vision integration and scalability.

Abstract

Inspired by foveal vision, hard attention models promise interpretability and parameter economy. However, existing models like the Recurrent Model of Visual Attention (RAM) and Deep Recurrent Attention Model (DRAM) failed to model the hierarchy of human vision system, that compromise on the visual exploration dynamics. As a result, they tend to produce attention that are either overly fixational or excessively saccadic, diverging from human eye movement behavior. In this paper, we propose a Multi-Level Recurrent Attention Model (MRAM), a novel hard attention framework that explicitly models the neural hierarchy of human visual processing. By decoupling the function of glimpse location generation and task execution in two recurrent layers, MRAM emergent a balanced behavior between fixation and saccadic movement. Our results show that MRAM not only achieves more human-like attention dynamics, but also consistently outperforms CNN, RAM and DRAM baselines on standard image classification benchmarks.

Emergence of Fixational and Saccadic Movements in a Multi-Level Recurrent Attention Model for Vision

TL;DR

The paper tackles the mismatch between current hard attention models and human visual exploration by introducing MRAM, a two-layer recurrent attention framework that decouples gaze control from final classification. The lower layer governs fast glimpse updates (saccade-like decisions) while the upper layer handles slow, high-level recognition, with a hybrid baseline stabilizing training through feedback-like signals. Experiments on MNIST, FashionMNIST, and FER2013 show MRAM not only matches or exceeds baseline accuracy but also yields emergent eye-movement patterns resembling human fixations and saccades, aligning with empirical eye-tracking data. The results suggest that hierarchical, biologically inspired attention can improve both interpretability and task performance, while highlighting areas for enhancement in peripheral vision integration and scalability.

Abstract

Inspired by foveal vision, hard attention models promise interpretability and parameter economy. However, existing models like the Recurrent Model of Visual Attention (RAM) and Deep Recurrent Attention Model (DRAM) failed to model the hierarchy of human vision system, that compromise on the visual exploration dynamics. As a result, they tend to produce attention that are either overly fixational or excessively saccadic, diverging from human eye movement behavior. In this paper, we propose a Multi-Level Recurrent Attention Model (MRAM), a novel hard attention framework that explicitly models the neural hierarchy of human visual processing. By decoupling the function of glimpse location generation and task execution in two recurrent layers, MRAM emergent a balanced behavior between fixation and saccadic movement. Our results show that MRAM not only achieves more human-like attention dynamics, but also consistently outperforms CNN, RAM and DRAM baselines on standard image classification benchmarks.
Paper Structure (18 sections, 2 equations, 4 figures, 3 tables)

This paper contains 18 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Recurrent Models of Visual Attention
  4. Hierarchical and Multi-Scale Extensions
  5. Method
  6. RAM
  7. MRAM
  8. Hybrid Baseline Estimation
  9. Training
  10. Experiments
  11. MNIST
  12. FashionMNIST
  13. Self-Emergence of Fixations and Saccades in Category Classification
  14. FER2013
  15. Discussion
  16. ...and 3 more sections

Figures (4)

  • Figure 1: The multi-layer recurrent attention model.
  • Figure 2: Comparison of learned policies on FashionMNIST. Red rectangles show the actual glimpse image served as input to the attention model. The top row corresponds to MRAM, the middle row corresponds to DRAM, and the bottom row corresponds to RAM.
  • Figure 3: Analysis of fixation duration distribution and saccade distance distribution using Kernel Distance Estimate (KDE). (a) shows the fixation duration distribution of all models in FashionMNIST. (b) shows the saccade distance distribution of all models in FashionMNIST.
  • Figure 4: Examples of the learned policy on the FER2013 classification task.