GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers

TL;DR

The paper addresses the mismatch between traditional positional encodings and the 3D geometry inherent in multi-view vision tasks. It introduces geometry-aware attention (GTA), which encodes token relationships through relative transformations $\rho_{g_i g_j^{-1}}$ applied to $K_j$ and $V_j$, with queries transformed into a shared coordinate frame to compute attention. GTA demonstrates consistent performance gains over state-of-the-art NVS transformers across synthetic and real datasets, achieving sharper reconstructions with faster learning and without additional learnable parameters. This geometry-guided approach enables more accurate 3D reasoning in transformer models, with practical implications for efficient 3D scene understanding from sparse views.

Abstract

As transformers are equivariant to the permutation of input tokens, encoding the positional information of tokens is necessary for many tasks. However, since existing positional encoding schemes have been initially designed for NLP tasks, their suitability for vision tasks, which typically exhibit different structural properties in their data, is questionable. We argue that existing positional encoding schemes are suboptimal for 3D vision tasks, as they do not respect their underlying 3D geometric structure. Based on this hypothesis, we propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation determined by the geometric relationship between queries and key-value pairs. By evaluating on multiple novel view synthesis (NVS) datasets in the sparse wide-baseline multi-view setting, we show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models without any additional learned parameters and only minor computational overhead.

Figures (27)

• Figure 1: GTA mechanism.$\rho^{-1}$ and $\rho^{\rm T}$ together take $Q,K$ and $V$ to a shared coordinate space, and the $\rho$ gets the attention output back to each token's coordinate space.
• Figure 2: Synthetic experiment. Left: Texture of the surrounding sphere. Right: Model architecture. The query pair consists of a learned constant value and a target extrinsic $g^*$.
• Figure 2: Test metrics. Left: CLEVR-TR, Right: MSN-Hard. $\dagger$Models are trained and evaluated on MultiShapeNet, not MSN-Hard. They are different but generated from the same distribution.
• Figure 3: Results on the synthetic dataset. Left: The solid and dashed lines indicate test and train errors. Right: Patches predicted with different PE schemes.
• Figure 3: Results on RealEstate10k and ACID. Top: NeRF methods. Bottom: transformer methods.