Towards Long-window Anchoring in Vision-Language Model Distillation

TL;DR

This work addresses the challenge of extending long-context understanding in small vision-language models. It introduces LAid, a distillation framework that transfers long-range attention from large VLMs to smaller students by combining head-level alignment with Fourier-based position distillation, preserving crucial low-frequency positional information. Empirical results on Visual HayStacks show up to 3.2× longer effective context while maintaining or improving standard VL benchmarks, with ablations confirming the necessity of the Fourier-enabled and head-alignment components. The study provides practical guidance for building more efficient long-context VLMs and offers theoretical insights into how positional understanding propagates across model scales.

Abstract

While large vision-language models (VLMs) demonstrate strong long-context understanding, their prevalent small branches fail on linguistics-photography alignment for a limited window size. We discover that knowledge distillation improves students' capability as a complement to Rotary Position Embeddings (RoPE) on window sizes (anchored from large models). Building on this insight, we propose LAid, which directly aims at the transfer of long-range attention mechanisms through two complementary components: (1) a progressive distance-weighted attention matching that dynamically emphasizes longer position differences during training, and (2) a learnable RoPE response gain modulation that selectively amplifies position sensitivity where needed. Extensive experiments across multiple model families demonstrate that LAid-distilled models achieve up to 3.2 times longer effective context windows compared to baseline small models, while maintaining or improving performance on standard VL benchmarks. Spectral analysis also suggests that LAid successfully preserves crucial low-frequency attention components that conventional methods fail to transfer. Our work not only provides practical techniques for building more efficient long-context VLMs but also offers theoretical insights into how positional understanding emerges and transfers during distillation.

Figures (4)

• Figure 1: Effective context window comparison. Left: The Visual Haystack task requiring retrieval from multi-image inputs. Right: Qwen2.5-VL accuracy across scales. Larger models (32B) sustain effective performance ($>$0.5) significantly longer than smaller counterparts (3B, 7B) despite identical architectures, revealing a scale-dependent RoPE awareness gap that our method targets.
• Figure 2: Overview of the LAid framework. (a) Position-aware Knowledge Transfer: LAid significantly extends the student's context length (purple) to approach the teacher's capability (gray), far exceeding the baseline (orange). (b) Fourier-Enhanced Position Distillation: Teacher attention heads capture positional information across frequency bands via mRoPE. We optimize weights ($w$) to distill these components into the student head, forming a rich Fourier series representation.
• Figure 3: Distillation performance comparison on Visual HayStack dataset. The bar chart illustrates the accuracy of Base, Knowledge Distillation (KD), and our LAid method across increasing haystack sizes (1 to 100 images). Using Qwen2.5-VL-7B as the student model and Qwen2.5-VL-32B as the teacher, LAid consistently outperforms both baseline and standard KD approaches.
• Figure 4: Head-level Knowledge Flow Analysis across different Visual HayStack Size. The leftmost column shows the student model's behavior, revealing a dramatic improvement from rapidly decaying activations (before LAid, top) to more stable patterns (after LAid, bottom). The middle columns display teachers' Local Position Heads (blue) with a slight downward trend across increasing context lengths, while the rightmost columns show Global Position Heads (orange) that maintain or slightly increase activation values at longer distances. LAid transfers this balanced position awareness to the student model, enabling it to maintain activation strength across the full range of context windows rather than focusing primarily on short-range dependencies.