Remember Me: Bridging the Long-Range Gap in LVLMs with Three-Step Inference-Only Decay Resilience Strategies

TL;DR

The paper tackles the problem of long-range attention decay in RoPE-based LVLMs, which impairs cross-modal reasoning over distant token pairs. It introduces Three-step Decay-Resilience Strategies (T-DRS), an inference-only framework consisting of SD-DRS, DC-DRS, and reRD-DRS, to reinforce distant dependencies while preserving locality, forming $A^{\text{T-DRS}} = A + A^{sd} + A^{dc} + A^{re}$. Evaluated in a training-free setting on ScienceQA-IMG, GQA, TextVQA, and POPE, T-DRS yields consistent improvements across multiple LVLM backbones (e.g., LLaVA1.5-7B, InterVL2-8B, Qwen2.5-VL-7B), demonstrating generality and robustness in long-context VQA tasks. Overall, the approach enables more reliable global coherence in multimodal reasoning without model retraining, offering a practical plug-and-play enhancement for Vision-Language understanding and QA systems.

Abstract

Large Vision-Language Models (LVLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they still face critical challenges in modeling long-range dependencies under the usage of Rotary Positional Encoding (ROPE). Although it can facilitate precise modeling of token positions, it induces progressive attention decay as token distance increases, especially with progressive attention decay over distant token pairs, which severely impairs the model's ability to remember global context. To alleviate this issue, we propose inference-only Three-step Decay Resilience Strategies (T-DRS), comprising (1) Semantic-Driven DRS (SD-DRS), amplifying semantically meaningful but distant signals via content-aware residuals, (2) Distance-aware Control DRS (DC-DRS), which can purify attention by smoothly modulating weights based on positional distances, suppressing noise while preserving locality, and (3) re-Reinforce Distant DRS (reRD-DRS), consolidating the remaining informative remote dependencies to maintain global coherence. Together, the T-DRS recover suppressed long-range token pairs without harming local inductive biases. Extensive experiments on Vision Question Answering (VQA) benchmarks demonstrate that T-DRS can consistently improve performance in a training-free manner. The code can be accessed in https://github.com/labixiaoq-qq/Remember-me

Figures (4)

• Figure 1: (A.) RoPE suffers from rapid attention decay over long token distances, weakening long-range dependencies between text and image tokens. (B.1, B.2, B.3) The proposed T-DRS framework alleviates this via three stages to collaboratively enhance and stabilize distant attention. (C.) The curves of the cross-attention weight decay in RoPE-like approaches, DC-DRS, and reRD-DRS, respectively.
• Figure 2: The proposed T-DRS is an inference-only framework. Given image and text inputs, visual and instruction features ($\textit{S}_{vision}$, $\textit{S}_{instr}$) can be extracted, then concatenated as $S$, input into the RoPE-attention architecture. The output attention logits $A$ experienced long-range dependencies decay, we then modulated three DRS: (1) SD-DRS ($A^{sd}$) amplifies semantically relevant distant tokens; (2) DC-DRS ($A^{dc}$) applies local smoothing technique, avoiding the local structure being perturbing, and (3) reRD-DRS ($A^{re}$) is sepecialized for re-strengthens long-range attention with rich semantics. Integrating the pre-softmax attention map, $A^{T-DRS}$ is used to generate the final output.
• Figure 3: Four hyperparameters are evaluated using ScienceQA-IMG and POPE datasets. The values of $\lambda$ are determined following the selection of $w_{min}$, and the two hyperparameters in reRD-DRS are configured subsequent to the determination of those in DC-DRS. We assign the value of three times $\left|{A}\right|_{min}$ and 1 to the two hyperparameters in DC-DRS, and twice of $\left|{A}\right|_{min}$ with set 0.8 for $\lambda_{re}$ in ScienceQA-IMG, and 1 for POPE in reRD-DRS.
• Figure 4: Visualization results of cross-attention of $S$ across different stages. (A) RoPE-only ($A$). (B) With SD-DRS and DC-DRS ($A+A^{sd}+A^{dc}$), attention begins to converge towards the central focus. (C) Under the full model setting ($A^{T-DRS}$), the model can further reinforce informative long-range connections.