Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Multimodal Positional Encoding in Vision-Language Models

Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang, Junyang Lin, Shuai Bai

TL;DR

This work systematically analyzes multimodal Rotary Position Embedding in Vision-Language Models, exposing design gaps in position encoding and frequency allocation that hinder cross-modal grounding and long-range modeling. It proposes three practical guidelines—positional coherence, full frequency utilization, and preservation of textual priors—leading to two plug-and-play solutions, Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), complemented by a spatial-reset mechanism. Across image, video, and grounding benchmarks, the proposed variants outperform existing RoPE methods, addressing both general and fine-grained multimodal understanding while maintaining compatibility with text-only RoPE for transfer learning. The work delivers actionable insights and code to advance robust, unified positional encoding for versatile VLMs.

Abstract

Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors-ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code will be avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.

Revisiting Multimodal Positional Encoding in Vision-Language Models

TL;DR

This work systematically analyzes multimodal Rotary Position Embedding in Vision-Language Models, exposing design gaps in position encoding and frequency allocation that hinder cross-modal grounding and long-range modeling. It proposes three practical guidelines—positional coherence, full frequency utilization, and preservation of textual priors—leading to two plug-and-play solutions, Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), complemented by a spatial-reset mechanism. Across image, video, and grounding benchmarks, the proposed variants outperform existing RoPE methods, addressing both general and fine-grained multimodal understanding while maintaining compatibility with text-only RoPE for transfer learning. The work delivers actionable insights and code to advance robust, unified positional encoding for versatile VLMs.

Abstract

Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors-ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code will be avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.
Paper Structure (32 sections, 6 equations, 5 figures, 7 tables)

This paper contains 32 sections, 6 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Analysis of Multimodal Rotary Position Embedding
  3. Preliminaries: Vanilla RoPE
  4. Position Design
  5. 1D Sequential Design.
  6. Multi-Dimensional Design.
  7. Towards an Optimal Position Design
  8. Frequency Allocation
  9. Frequency Allocation in 1D RoPE.
  10. Multi-Dimensional Frequency Allocation.
  11. Towards an Optimal Frequency Allocation
  12. Compatibility with Text-only RoPE.
  13. Proposed Multimodal RoPEs
  14. Experiment
  15. Experimental Setup
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Position design of different multimodal RoPE variants. The illustrated example follows an interleaved multimodal sequence: <system prompt>, <video 1>, <text>, <image 1>, <text>, <image 2>, <text>, <image 3>, <text>, <generated text>.
  • Figure 2: Visual attention sink in MRoPE. Average attention scores to input sequence from the ChartQA (Left) and VideoMME-short (Right).
  • Figure 3: Frequence allocation of different multimodal RoPEs.
  • Figure 4: The long-range decay property of MRoPE, MHRoPE and MRoPE-I.
  • Figure 5: Video extrapolation performance. Models are trained with a context length of 32k (256 frames) and extrapolated to 64k (512 frames), 128k (1024 frames), and 256k (2048 frames).