Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning

Jian Shi, Michael Birsak, Wenqing Cui, Zhenyu Li, Peter Wonka

TL;DR

The paper addresses how positional embeddings govern geometry in vision transformers, arguing that PEs act as a learned spatial kernel that can both help and hinder spatial reasoning across views. It introduces token-level diagnostics to quantify multi-view geometric consistency and demonstrates, across 14 foundation ViTs, that geometry is driven by PE consistency rather than content alone. The authors propose a training-free token re-indexing to restore PE alignment and analyze VGGT to show how multi-view aggregation can implicitly infer a canonical coordinate system. The findings reveal PEs as a causal mechanism shaping spatial structure in ViTs, with practical implications for designing robust multi-view vision models.

Abstract

This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes

Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning

TL;DR

The paper addresses how positional embeddings govern geometry in vision transformers, arguing that PEs act as a learned spatial kernel that can both help and hinder spatial reasoning across views. It introduces token-level diagnostics to quantify multi-view geometric consistency and demonstrates, across 14 foundation ViTs, that geometry is driven by PE consistency rather than content alone. The authors propose a training-free token re-indexing to restore PE alignment and analyze VGGT to show how multi-view aggregation can implicitly infer a canonical coordinate system. The findings reveal PEs as a causal mechanism shaping spatial structure in ViTs, with practical implications for designing robust multi-view vision models.

Abstract

This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes
Paper Structure (28 sections, 12 equations, 3 figures, 4 tables)

This paper contains 28 sections, 12 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Geometry and Locality in ViTs
  4. The Role of Positional Embedding
  5. PEs as Spatial Kernels
  6. Background: Absolute, Relative, and Rotary PEs
  7. Absolute positional encodings.
  8. Relative positional encodings.
  9. Rotary positional encodings (RoPE).
  10. A Kernel Interpretation of PEs
  11. Absolute & Relative Positional Encoding (RPE).
  12. Rotary positional encodings (RoPE).
  13. The PE-induced Kernel.
  14. Content vs. Position: When Geometry Breaks in ViTs
  15. ViT Representations Fail Under Identical Content
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Epipolar peak response visualization.Top-left: an example stereo pair and the corresponding SAM feature maps with and without PEs. Removing PE substantially weakens the spatial anchoring in the token representations. Remaining panels show the epipolar peak responses for each evaluated model. With PEs enabled, nearly all models (except BEiT) exhibit well-localized epipolar peaks, indicating preserved spatial anchoring. After removing PEs, epipolar responses vanish for most architectures, revealing that spatial anchoring degrades without positional signals. We use AnyUp wimmer2025anyup to obtain semantically meaningful feature visualizations. Please refer to the supplementary for (pairwise) shuffled PE visualizations.
  • Figure 2: Illustration of token re-indexing. The upper and bottom represent two feature maps that contain overlapping regions (in pixel space, annotated in red color) along the horizontal axis.
  • Figure :