Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models

TL;DR

The paper addresses the heavy compute burden of vision-language models and proposes a principled framework to identify layer- and token-level redundancy that enables skipping. It develops an information- and learning-theoretic approach to define functional and informational redundancy, and links these to geometric and cross-attention measures to locate skip-viable layers. Empirical validation across multiple VLMs shows that skipping layers with high redundancy can accelerate inference with minimal accuracy loss, while skipping non-redundant layers degrades performance. By connecting redundancy concepts to cross-attention analysis and partial information decomposition, the work provides a unified basis for several efficient inference techniques in VLMs.

Abstract

Vision-language models (VLMs) achieve incredible performance across a wide range of tasks, but their large size makes inference costly. Recent work shows that selectively skipping VLM layers can improve efficiency with minimal performance loss or even performance improvements. However, this technique remains underused due to the limited understanding of when layer skipping is beneficial. In this paper, we develop a framework that uses information and learning theory to characterize the conditions under which layer skipping enhances efficiency without sacrificing performance. Motivated by these observations, we analyze the evolution of the VLM's hidden representations through the LLM backbone and show that layers with large redundancy as predicted by our framework coincide with those skipped by popular layer-skipping methods in practice, providing a unified theoretical scaffolding for multiple efficient inference techniques. Our experiments demonstrate that skipping such layers yields faster inference that preserves performance, and also show that applying skipping outside these conditions leads to model degradation.

Figures (12)

• Figure 1: Early Exit and Late Entry. Specifically, the visual tokens are not passed into the first few layers but instead, directly inserted along with the prompt to the chosen layer for insertion or the vision tokens are removed from the forward pass after a certain layer.
• Figure 2: Implication relationships among different notions of redundancy.
• Figure 3: Visual Attention Ratio (VAR) JiangCZLSY2025 with respect to the answer text token for the LLaVA 7B and 13B. One can see that the majority of the cross-attention is isolated to the middle layers, and the early and late layers have minimal cross-attention.
• Figure 4: Empirical geometric and proximal redundancy experiments across layers for the LLaVA 1.5 7B/13B and LLaVA NeXT 7B. Across all the General VQA task (see Table \ref{['table:datasets']}) and models, the early layer vision tokens have low adjacent token cosine distances, and the textual and visual tokens have low adjacent token cosine distances in later layers.
• Figure 5: Empirical Geometric and Proximal Redundancy Experiments across layers for the Qwen 2.5 VL and Deepseek VL 7B VLMs. Across all datasets in the General VQA task (see Table \ref{['table:datasets']}) and models, the early layer vision tokens have low adjacent token cosine distances, and the textual and visual tokens have low adjacent token cosine distances in later layers.

Theorems & Definitions (36)

• Definition 1: Geometric $\epsilon$-redundancy
• Definition 2: $t$-proximal with probability $1-\epsilon$; Proximal Redundancy
• Definition 3: Functional $\epsilon$-redundancy
• Definition 4: Informational $\epsilon$-redundancy
• Theorem 1
• proof
• Theorem 2
• proof
• Proposition 1
• proof