Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GridPE: Unifying Positional Encoding in Transformers with a Grid Cell-Inspired Framework

Boyang Li, Yulin Wu, Nuoxian Huang, Wenjia Zhang

TL;DR

A novel positional encoding scheme inspired by Fourier analysis and the latest findings in computational neuroscience regarding grid cells is introduced, termed GridPE, which provides a unifying framework for positional encoding in arbitrary high-dimensional spaces.

Abstract

Understanding spatial location and relationships is a fundamental capability for modern artificial intelligence systems. Insights from human spatial cognition provide valuable guidance in this domain. Neuroscientific discoveries have highlighted the role of grid cells as a fundamental neural component for spatial representation, including distance computation, path integration, and scale discernment. In this paper, we introduce a novel positional encoding scheme inspired by Fourier analysis and the latest findings in computational neuroscience regarding grid cells. Assuming that grid cells encode spatial position through a summation of Fourier basis functions, we demonstrate the translational invariance of the grid representation during inner product calculations. Additionally, we derive an optimal grid scale ratio for multi-dimensional Euclidean spaces based on principles of biological efficiency. Utilizing these computational principles, we have developed a Grid-cell inspired Positional Encoding technique, termed GridPE, for encoding locations within high-dimensional spaces. We integrated GridPE into the Pyramid Vision Transformer architecture. Our theoretical analysis shows that GridPE provides a unifying framework for positional encoding in arbitrary high-dimensional spaces. Experimental results demonstrate that GridPE significantly enhances the performance of transformers, underscoring the importance of incorporating neuroscientific insights into the design of artificial intelligence systems.

GridPE: Unifying Positional Encoding in Transformers with a Grid Cell-Inspired Framework

TL;DR

A novel positional encoding scheme inspired by Fourier analysis and the latest findings in computational neuroscience regarding grid cells is introduced, termed GridPE, which provides a unifying framework for positional encoding in arbitrary high-dimensional spaces.

Abstract

Understanding spatial location and relationships is a fundamental capability for modern artificial intelligence systems. Insights from human spatial cognition provide valuable guidance in this domain. Neuroscientific discoveries have highlighted the role of grid cells as a fundamental neural component for spatial representation, including distance computation, path integration, and scale discernment. In this paper, we introduce a novel positional encoding scheme inspired by Fourier analysis and the latest findings in computational neuroscience regarding grid cells. Assuming that grid cells encode spatial position through a summation of Fourier basis functions, we demonstrate the translational invariance of the grid representation during inner product calculations. Additionally, we derive an optimal grid scale ratio for multi-dimensional Euclidean spaces based on principles of biological efficiency. Utilizing these computational principles, we have developed a Grid-cell inspired Positional Encoding technique, termed GridPE, for encoding locations within high-dimensional spaces. We integrated GridPE into the Pyramid Vision Transformer architecture. Our theoretical analysis shows that GridPE provides a unifying framework for positional encoding in arbitrary high-dimensional spaces. Experimental results demonstrate that GridPE significantly enhances the performance of transformers, underscoring the importance of incorporating neuroscientific insights into the design of artificial intelligence systems.
Paper Structure (15 sections, 23 equations, 4 figures, 1 table)

This paper contains 15 sections, 23 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Derivation of Grid Cell Positional Encoding
  4. Inner Product Translational Invariance
  5. Optimal Ratio of Grid Scale under Economy Principle
  6. Proposal of Grid-cell Positional Encoding (GridPE)
  7. Representing Encodings as Rotation Matrices
  8. Incorporating Real and Imaginary Parts into Information Vectors
  9. Preserving the Complex Form
  10. Feature Mapping of Positional Encoding Using Deep Learning
  11. Experiments
  12. Property of Distance Decay
  13. Performance of GridPE on Visual Task
  14. Results and Analysis
  15. Conclusion

Figures (4)

  • Figure 1: Illustration of VCO theory. The black neurons are presynaptic neurons with simple oscillation in their dendrites. The orange neuron is the grid cell, whose grid firing pattern results from the summation of the three oscillation with different preferred direction.
  • Figure 2: Illastration of relationship between firing fields diameter and periodicity. For two grid cells with adjacent discrete scale, the periodcity of the smaller scale must be less than the diameter of the larger one, otherwise the system cannot distinguish two locations.
  • Figure 3: The inner products intensity with the core points $(0.5, 0.5)$ of points in the unit square area. Left: the distribution of inner products is shown as a surface plot across the area. Right: a profile plot showing the inner products between core points and points with fixed $x=0.5$.
  • Figure 4: Comparison of Validation Accuracy over Epochs for ViT and PVT Models.