Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models

TL;DR

The paper addresses whether all layers in pretrained Vision-Language Models contribute to downstream tasks and finds evidence of Task-Interfering Layers whose removal improves performance on certain tasks. It introduces Task-Layer Interaction Vectors to quantify per-layer sensitivity and TaLo, a training-free test-time knockout that selects and bypasses interfering layers for each task, without updating parameters. Empirical results across multiple models and benchmarks show that Task-Interfering Layers are common, yet their interference patterns align with task similarity, enabling robust inference-time adaptation. TaLo delivers significant, training-free gains across multiple models and benchmarks, demonstrating practical inference-time adaptation and revealing modular organization in cross-modal reasoning. The work also outlines future directions for understanding layer-level task decomposition and potential training-time regularization guided by the Task-Layer framework.

Abstract

Current VLMs have demonstrated capabilities across a wide range of multimodal tasks. Typically, in a pretrained VLM, all layers are engaged by default to make predictions on downstream tasks. We find that intervening on a single layer, such as by zeroing its parameters, can improve the performance on certain tasks, indicating that some layers hinder rather than help downstream tasks. We systematically investigate how individual layers influence different tasks via layer intervention. Specifically, we measure the change in performance relative to the base model after intervening on each layer and observe improvements when bypassing specific layers. This improvement can be generalizable across models and datasets, indicating the presence of Task-Interfering Layers that harm downstream tasks' performance. We introduce Task-Layer Interaction Vector, which quantifies the effect of intervening on each layer of a VLM given a task. These task-interfering layers exhibit task-specific sensitivity patterns: tasks requiring similar capabilities show consistent response trends under layer interventions, as evidenced by the high similarity in their task-layer interaction vectors. Inspired by these findings, we propose TaLo (Task-Adaptive Layer Knockout), a training-free, test-time adaptation method that dynamically identifies and bypasses the most interfering layer for a given task. Without parameter updates, TaLo improves performance across various models and datasets, including boosting Qwen-VL's accuracy on the Maps task in ScienceQA by up to 16.6%. Our work reveals an unexpected form of modularity in pretrained VLMs and provides a plug-and-play, training-free mechanism to unlock hidden capabilities at inference time. The source code will be publicly available.

Figures (20)

• Figure 1: Overview of the task-interfering layer phenomenon. Each axis corresponds to a task category: AR (Attribute Reasoning), RR (Relation Reasoning), LR (Logical Reasoning), CP (Coarse Perception), FP-S (Fine-grained Perception [single-instance]), and FP-C (Fine-grained Perception [cross-instance]). Each plot shows model performance after zeroing out a single layer (solid curves), with the orange dashed line indicating the baseline performance (no intervention). In several tasks, performance improves upon layer removal, providing direct evidence for the existence of Task-Interfering Layers.
• Figure 2: Empirical Validation of the Task-Interfering Layers.(a) Visualization of the percentage change in accuracy across tasks after zeroing each layer on LLaVA-Next-LLaMA3-8B. Red indicates performance improvements relative to the base model, while blue indicates degradation. Many tasks show performance gains under layer interventions, indicating that interfering layers are commonly exist in VLMs. (b) The t-SNE visualization of task clusters. Each point in the figure represents a task encoded as a Task-Layer Interaction Vector $\mathbf{v}^{(\mathcal{T})}$. Tasks are clustered based on their pairwise similarities measured by Pearson correlation $\rho_{ij}$ between different vectors, with tasks requiring similar capabilities forming coherent clusters. The color-coded clusters correspond to different types of tasks, indicating that tasks with shared cognitive demands exhibit similar intervention responses, reflecting a structured functional layout in LLaVA li2024llavanext-strong (See Table \ref{['tab:clusters_tasks']} for complete clustering details).
• Figure 3: Framework of TaLo. TaLo first dynamically selects the Task-Interfering layer for a specific task and knocks out that layer in the final evaluation procedure.
• Figure 4: Consistency analysis of different interventions.
• Figure 5: Qualitative case study on random noise intervention.