Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation

Yiming Ren, Yujiu Yang, Junjie Wang

Abstract

Supervised fine-tuning (SFT) on visual instruction data often improves perceptual capabilities in vision-language models (VLMs) while degrading reasoning performance, creating a persistent reasoning tax during post-training. We investigate whether this degradation is related to disrupted access to depth-wise representations, and find that even fixed cross-depth aggregation substantially restores reasoning, suggesting that preserved cross-depth access is an important missing factor in VLM fine-tuning. Building on this observation, we propose Input-Adaptive Depth Aggregation (IADA), a lightweight mechanism that makes cross-depth retrieval input-adaptive, modality-aware, and efficiently parameterized through a low-rank bottleneck. On Qwen3-VL-2B, IADA improves the average reasoning score by 9.5 points and the average perception score by $3.3$ points over LoRA-only fine-tuning with only 0.14M additional parameters, with the strongest gains appearing in parameter-efficient low-rank settings.

Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation

Abstract

Supervised fine-tuning (SFT) on visual instruction data often improves perceptual capabilities in vision-language models (VLMs) while degrading reasoning performance, creating a persistent reasoning tax during post-training. We investigate whether this degradation is related to disrupted access to depth-wise representations, and find that even fixed cross-depth aggregation substantially restores reasoning, suggesting that preserved cross-depth access is an important missing factor in VLM fine-tuning. Building on this observation, we propose Input-Adaptive Depth Aggregation (IADA), a lightweight mechanism that makes cross-depth retrieval input-adaptive, modality-aware, and efficiently parameterized through a low-rank bottleneck. On Qwen3-VL-2B, IADA improves the average reasoning score by 9.5 points and the average perception score by points over LoRA-only fine-tuning with only 0.14M additional parameters, with the strongest gains appearing in parameter-efficient low-rank settings.

Paper Structure

This paper contains 25 sections, 13 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Fixed Depth Aggregation as a Diagnostic Baseline
  5. Why Fixed Aggregation Is Insufficient
  6. Input-Adaptive Depth Aggregation
  7. Relation to PEFT
  8. Experiments
  9. Experimental Setup
  10. Main Results: The Reasoning Tax and Its Recovery
  11. Ablations and Design Analysis
  12. From Fixed Aggregation to Adaptive Aggregation
  13. Interaction with LoRA Rank
  14. Query Design and Modality Conditioning
  15. Low-Rank Bottleneck
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Overview of IADA. Compared with LoRA fine-tuning in (a), IADA in (b) inserts an IADA block at block boundaries to enable explicit cross-depth routing. IADA generates modality-specific queries from the current hidden states, retrieves information from cached states at previous $n_d$ block boundaries, and injects the resulting aggregated residuals back through gated residual addition.