Enhancing Multi-Image Understanding through Delimiter Token Scaling

TL;DR

This work addresses cross-image information leakage in large vision–language models by analyzing image delimiter tokens and proposing a simple, training-free remedy. By scaling the hidden states of delimiter tokens with a factor $\lambda>1$, the method strengthens image-wise tagging and concentrates attention within each image, substantially reducing cross-image interference while preserving intra-image interactions. The approach yields consistent improvements across four multi-image benchmarks (Mantis, MuirBench, MIRB, QBench2) and across multi-document/multi-table tasks (MultiNews, WCEP-10, TQABench) without adding training or inference costs, and remains compatible with optimized attention kernels like FlashAttention. The results demonstrate broad generality across model sizes and modalities, offering a practical, efficient enhancement for multi-instance reasoning in LVLMs with real-world impact on cross-image understanding and downstream tasks.

Abstract

Large Vision-Language Models (LVLMs) achieve strong performance on single-image tasks, but their performance declines when multiple images are provided as input. One major reason is the cross-image information leakage, where the model struggles to distinguish information across different images. Existing LVLMs already employ delimiter tokens to mark the start and end of each image, yet our analysis reveals that these tokens fail to effectively block cross-image information leakage. To enhance their effectiveness, we propose a method that scales the hidden states of delimiter tokens. This enhances the model's ability to preserve image-specific information by reinforcing intra-image interaction and limiting undesired cross-image interactions. Consequently, the model is better able to distinguish between images and reason over them more accurately. Experiments show performance gains on multi-image benchmarks such as Mantis, MuirBench, MIRB, and QBench2. We further evaluate our method on text-only tasks that require clear distinction. The method improves performance on multi-document and multi-table understanding benchmarks, including TQABench, MultiNews, and WCEP-10. Notably, our method requires no additional training or inference cost.

Figures (18)

• Figure 1: Impact of image delimiter tokens on attention maps. (a) With delimiter tokens, clear triangular patterns mark image boundaries. (b) Without them, these patterns disappear. (c) Replacing them with other special tokens (<|im_start|>) yields the same effect.
• Figure 2: (a) Attention to the second-image delimiter. (b) Image tagging values.
• Figure 3: Effect of scaling image delimiter tokens on attention. Left: attention computation flow in a transformer block. Right: before scaling (top), delimiter tokens receive limited attention, leading to cross-image leakage. After scaling (bottom), delimiter tokens act as strong attractors (like sink tokens), distinguishing between images while preserving intra-image interactions (Property 2).
• Figure 4: Qualitative comparison of (a) attention maps before and after applying our method, and (b) zoomed-in views of the red boxes. After applying our method, cross-image interaction is reduced.
• Figure 5: (a) Inter-image interaction changes before and after scaling (normalized to 1.00 before scaling). (b) After scaling, Image 3 tokens receive the strongest attention from the third delimiter token.