Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Sensorimotor Vision Transformer

Konrad Gadzicki, Kerstin Schill, Christoph Zetzsche

TL;DR

The paper tackles the computational and memory inefficiency of Vision Transformers by introducing the Sensorimotor Transformer (SMT), which mimics human saccadic sampling to focus on high-information image regions. It predicts salient patches using i2D cues derived from a Ratio-of-Gaussians contrast operator and a curvature-based 2D feature detector, then processes only the top $m$ patches with a ViT backbone. SMT demonstrates competitive ImageNet-1k accuracy while significantly reducing memory usage, with memory scaling primarily with the number of selected patches. This biologically inspired approach offers a practical, resource-efficient alternative for image analysis and provides insights into integrating sensorimotor principles with transformer architectures for constrained environments.

Abstract

This paper presents the Sensorimotor Transformer (SMT), a vision model inspired by human saccadic eye movements that prioritize high-saliency regions in visual input to enhance computational efficiency and reduce memory consumption. Unlike traditional models that process all image patches uniformly, SMT identifies and selects the most salient patches based on intrinsic two-dimensional (i2D) features, such as corners and occlusions, which are known to convey high-information content and align with human fixation patterns. The SMT architecture uses this biological principle to leverage vision transformers to process only the most informative patches, allowing for a substantial reduction in memory usage that scales with the sequence length of selected patches. This approach aligns with visual neuroscience findings, suggesting that the human visual system optimizes information gathering through selective, spatially dynamic focus. Experimental evaluations on Imagenet-1k demonstrate that SMT achieves competitive top-1 accuracy while significantly reducing memory consumption and computational complexity, particularly when a limited number of patches is used. This work introduces a saccade-like selection mechanism into transformer-based vision models, offering an efficient alternative for image analysis and providing new insights into biologically motivated architectures for resource-constrained applications.

A Sensorimotor Vision Transformer

TL;DR

The paper tackles the computational and memory inefficiency of Vision Transformers by introducing the Sensorimotor Transformer (SMT), which mimics human saccadic sampling to focus on high-information image regions. It predicts salient patches using i2D cues derived from a Ratio-of-Gaussians contrast operator and a curvature-based 2D feature detector, then processes only the top patches with a ViT backbone. SMT demonstrates competitive ImageNet-1k accuracy while significantly reducing memory usage, with memory scaling primarily with the number of selected patches. This biologically inspired approach offers a practical, resource-efficient alternative for image analysis and provides insights into integrating sensorimotor principles with transformer architectures for constrained environments.

Abstract

This paper presents the Sensorimotor Transformer (SMT), a vision model inspired by human saccadic eye movements that prioritize high-saliency regions in visual input to enhance computational efficiency and reduce memory consumption. Unlike traditional models that process all image patches uniformly, SMT identifies and selects the most salient patches based on intrinsic two-dimensional (i2D) features, such as corners and occlusions, which are known to convey high-information content and align with human fixation patterns. The SMT architecture uses this biological principle to leverage vision transformers to process only the most informative patches, allowing for a substantial reduction in memory usage that scales with the sequence length of selected patches. This approach aligns with visual neuroscience findings, suggesting that the human visual system optimizes information gathering through selective, spatially dynamic focus. Experimental evaluations on Imagenet-1k demonstrate that SMT achieves competitive top-1 accuracy while significantly reducing memory consumption and computational complexity, particularly when a limited number of patches is used. This work introduces a saccade-like selection mechanism into transformer-based vision models, offering an efficient alternative for image analysis and providing new insights into biologically motivated architectures for resource-constrained applications.

Paper Structure

This paper contains 19 sections, 10 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Key Contributions
  3. Related Work
  4. Vision Transformer
  5. Models of Saccadic Eye Movements
  6. Model and Methods
  7. Connection between Sensorimotor Systems and Transformers
  8. System Overview
  9. Prediction of Salient Regions
  10. Image contrast
  11. Curvature Operator
  12. Filter Design for i2D Selectivity
  13. Curvature Computation
  14. Patch selection
  15. Model Implementation, Parameters, and Training
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Overview of the proposed Sensorimotor Transformer
  • Figure 2: Ratio of Gaussians contrast operator.
  • Figure 3: Polar separable filter functions. \ref{['fig:s_n']} and \ref{['fig:c_n']} show the angular filter functions and \ref{['fig:laplace']} the radial one.
  • Figure 4: Filter responses of the curvature components \ref{['fig:laplace_response']} and \ref{['fig:ecc_response']} and the final curvature \ref{['fig:D_response']}.
  • Figure 5: Top 1 accuracy.