LOOPE: Learnable Optimal Patch Order in Positional Embeddings for Vision Transformers
Md Abtahi Majeed Chowdhury, Md Rifat Ur Rahman, Akil Ahmad Taki
TL;DR
This work addresses the challenge of effectively encoding 2D spatial structure in Vision Transformers by tackling patch ordering in positional embeddings. It introduces LOOPE, a Learnable Optimal Patch Order method that combines a static Gilbert-order $X_G$ with a learnable context bias $X_C$ to produce a unified patch sequence $X= X_G+X_C$, used in a frequency-based embedding $E(X)=\sin(XW^T)|\cos(XW^T)$. To evaluate the structural integrity of embeddings, the authors propose the Three-Cell Benchmark and PESI metrics (Undirected Monotonicity $M_U$, Directed Monotonicity $M_D$, Undirected Asymmetry $A_{SU}$). Across multiple ViT architectures and datasets, LOOPE yields consistent accuracy gains and provides a richer diagnostic than traditional PE assessments, highlighting the importance of both global locality and context-driven ordering in 2D-to-1D spatial representations. The work offers a principled framework for designing and evaluating positional embeddings in vision models, with implications for robustness and generalization across resolutions and tasks.
Abstract
Positional embeddings (PE) play a crucial role in Vision Transformers (ViTs) by providing spatial information otherwise lost due to the permutation invariant nature of self attention. While absolute positional embeddings (APE) have shown theoretical advantages over relative positional embeddings (RPE), particularly due to the ability of sinusoidal functions to preserve spatial inductive biases like monotonicity and shift invariance, a fundamental challenge arises when mapping a 2D grid to a 1D sequence. Existing methods have mostly overlooked or never explored the impact of patch ordering in positional embeddings. To address this, we propose LOOPE, a learnable patch-ordering method that optimizes spatial representation for a given set of frequencies, providing a principled approach to patch order optimization. Empirical results show that our PE significantly improves classification accuracy across various ViT architectures. To rigorously evaluate the effectiveness of positional embeddings, we introduce the "Three Cell Experiment", a novel benchmarking framework that assesses the ability of PEs to retain relative and absolute positional information across different ViT architectures. Unlike standard evaluations, which typically report a performance gap of 4 to 6% between models with and without PE, our method reveals a striking 30 to 35% difference, offering a more sensitive diagnostic tool to measure the efficacy of PEs. Our experimental analysis confirms that the proposed LOOPE demonstrates enhanced effectiveness in retaining both relative and absolute positional information.