Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding

TL;DR

This work addresses the misalignment between visual token positions and instruction tokens in vision-language models caused by raster-scan encoding and Rotary Position Embedding (RoPE) decay. It introduces Pyramid-descent Visual Position Encoding (PyPE), which repositions visual tokens in a 2D grid from periphery to center and incrementally expands the central receptive field across layers, enabling multi-granularity perception and reducing reliance on anchor tokens. Extensive experiments across visual question answering and general multimodal benchmarks demonstrate that PyPE consistently improves perception and reasoning capabilities across model sizes, with qualitative analyses showing reduced hallucinations and more continuous attention. The approach is simple, generalizable, and has practical impact for enhancing cross-modal understanding in VLMs, with code available for replication.

Abstract

Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models' comprehensive perception performance across different levels of granularity. In this work, we propose Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. By assigning visual position indexes from the periphery to the center and expanding the central receptive field incrementally, PyPE addresses the limitations of traditional raster-scan methods and mitigates the long-term decay effects induced by Rotary Position Embedding (RoPE). Our method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements and countering the over-reliance on anchor tokens. Extensive experimental evaluations demonstrate that PyPE consistently improves the general capabilities of VLMs across various sizes. Code is available at https://github.com/SakuraTroyChen/PyPE.

Figures (17)

• Figure 1: Layer-wise attention visualization of visual-to-instruction information flow. Displayed from top to bottom are the attention heatmaps from LLaVA-1.5-7B trained with raster-scan and concentric PE, respectively. The example is derived from LLaVA-Bench liu2024visual and the query is "Describe this photo in detail".
• Figure 2: An overview of patch indexes and corresponding causal mask from raster-scan, concentric, and All-One position encoding on an example from COCO lin2014microsoft.
• Figure 3: An overview of the proposed PyPE. We first reorganize the visual tokens from their vanilla flattened 1D sequence form into the 2D format. Subsequently, we assign visual position indexes from the periphery to the center and expand the central receptive field incrementally across the layers with an interval of $t$.
• Figure 4: Illustration of the multi-granularity perception capability of PyPE with a sample from LLaVA-Bench. The case study is based on LLaVA-1.5-7B and the query is "Describe this photo in detail". The misunderstandings and hallucinations of visual contents are highlighted in red. We also provide a corresponding layer-wise attention visualization of PyPE, with the heatmap arranged from the upper left to the lower right, indicating layers 1 to 32.
• Figure 5: Visualization of anchor tokens in baselines and PyPE.