Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Positional Preservation Embedding for Multimodal Large Language Models

Mouxiao Huang, Borui Jiang, Dehua Zheng, Hailin Hu, Kai Han, Xinghao Chen

TL;DR

The paper tackles inefficiencies in multimodal large language models caused by dense visual tokens and disrupted spatiotemporal layouts during compression. It introduces Positional Preservation Embedding (PPE), a parameter-free encoding operator that splits and preserves multiple positional IDs per merged token, enabling cascade compression across Transformer layers. Empirical results on MMBench, TextVQA, and VideoMME show consistent improvements (roughly $2\%\sim 5\%$) and substantial token reductions (55% spatial and up to 90% with cascade), while maintaining or improving task performance. PPE’s plug-and-play compatibility with existing token merging methods and its emphasis on preserving positional cues highlight its practical impact for efficient and effective vision-language reasoning.

Abstract

Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing token merging methods reduce sequence length but frequently disrupt spatial layouts and temporal continuity by disregarding positional relationships. In this work, we propose a novel encoding operator dubbed as \textbf{P}ositional \textbf{P}reservation \textbf{E}mbedding (\textbf{PPE}), which has the main hallmark of preservation of spatiotemporal structure during visual token compression. PPE explicitly introduces the disentangled encoding of 3D positions in the token dimension, enabling each compressed token to encapsulate different positions from multiple original tokens. Furthermore, we show that PPE can effectively support cascade clustering -- a progressive token compression strategy that leads to better performance retention. PPE is a parameter-free and generic operator that can be seamlessly integrated into existing token merging methods without any adjustments. Applied to state-of-the-art token merging framework, PPE achieves consistent improvements of $2\%\sim5\%$ across multiple vision-language benchmarks, including MMBench (general vision understanding), TextVQA (layout understanding) and VideoMME (temporal understanding). These results demonstrate that preserving positional cues is critical for efficient and effective MLLM reasoning.

Positional Preservation Embedding for Multimodal Large Language Models

TL;DR

The paper tackles inefficiencies in multimodal large language models caused by dense visual tokens and disrupted spatiotemporal layouts during compression. It introduces Positional Preservation Embedding (PPE), a parameter-free encoding operator that splits and preserves multiple positional IDs per merged token, enabling cascade compression across Transformer layers. Empirical results on MMBench, TextVQA, and VideoMME show consistent improvements (roughly ) and substantial token reductions (55% spatial and up to 90% with cascade), while maintaining or improving task performance. PPE’s plug-and-play compatibility with existing token merging methods and its emphasis on preserving positional cues highlight its practical impact for efficient and effective vision-language reasoning.

Abstract

Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing token merging methods reduce sequence length but frequently disrupt spatial layouts and temporal continuity by disregarding positional relationships. In this work, we propose a novel encoding operator dubbed as \textbf{P}ositional \textbf{P}reservation \textbf{E}mbedding (\textbf{PPE}), which has the main hallmark of preservation of spatiotemporal structure during visual token compression. PPE explicitly introduces the disentangled encoding of 3D positions in the token dimension, enabling each compressed token to encapsulate different positions from multiple original tokens. Furthermore, we show that PPE can effectively support cascade clustering -- a progressive token compression strategy that leads to better performance retention. PPE is a parameter-free and generic operator that can be seamlessly integrated into existing token merging methods without any adjustments. Applied to state-of-the-art token merging framework, PPE achieves consistent improvements of across multiple vision-language benchmarks, including MMBench (general vision understanding), TextVQA (layout understanding) and VideoMME (temporal understanding). These results demonstrate that preserving positional cues is critical for efficient and effective MLLM reasoning.
Paper Structure (27 sections, 7 equations, 6 figures, 9 tables)

This paper contains 27 sections, 7 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Visual Token Merging
  5. Positional Encoding in MLLMs
  6. Methodology
  7. Preliminary
  8. Positional Preservation Embedding
  9. Cascade Compression with PPE
  10. Experiment
  11. Experimental Setup
  12. Main Results
  13. Cascade Compression
  14. Ablation Study
  15. Insight into why PPE better preserves token semantics.
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Comparison between PPE and other token merging methods of processing positional IDs. To simplify, the components such as the visual encoder are omitted. (a) ChatUniVi jin2023chatunivi mainly assigns randomize ID value to the clustered visual tokens. (b) PACT dhouib2025pactpruningclusteringbasedtoken retains the ID of the cluster center for the clustered visual tokens. (c) Proposed PPE splits the IDs of compressed token on different dimensions, so that each compressed token could contain several original position IDs.
  • Figure 2: The overview pipeline of the proposed PPE with cascade compression.Left: Main idea of Positional Preservation Embedding (PPE) integrated in token compression. For each RoPE ID in compressed token embedding, PPE splits the dimension into chunks to prefill multiple position IDs. The IDs of tokens with high importance scores are reserved preferentially. Right: The MLLM pipeline integrating PPE and cascade compression. Token compression is applied in multiple layers, each with PPE. See main text for more explanation.
  • Figure 3: Attention statistics and visualizations of samples from TextVQA. (a--b) Quantitative comparison of entropy and variance. (c--d) Qualitative attention score visualizations of case 34602.
  • Figure 4: Qualitative examples for the image captioning task, from Flickr30k benchmark.
  • Figure 5: Qualitative comparison of attention visualizations. Each row corresponds to a different sample: (a, d, g) original input, (b, e, h) Chat-UniVi outputs, and (c, f, i) PPE outputs.
  • ...and 1 more figures