Towards Minimal Fine-Tuning of VLMs

TL;DR

Image-LoRA presents a vision-focused PEFT strategy for transformer-based VLMs by restricting adaptations to the visual-token span and a small, diverse set of attention heads, stabilized by selection-size normalization. The method achieves substantial reductions in adapter-only training FLOPs and trainable parameters while preserving pure-text reasoning and performing competitively on grounded visual tasks and VQA with respect to standard LoRA. A first-order head-importance proxy enables efficient, one-shot head selection, and experiments across text-heavy to image-heavy regimes demonstrate favorable accuracy-efficiency trade-offs, especially in image-rich settings. Limitations include scalability, batching challenges, and architectural dependence on explicit visual-token representations, suggesting avenues for further scaling and analysis of attention-pattern shifts. Overall, Image-LoRA offers a structured, efficient pathway to fine-tune VLMs for grounded visual reasoning without compromising language-focused capabilities.

Abstract

We introduce Image-LoRA, a lightweight parameter efficient fine-tuning (PEFT) recipe for transformer-based vision-language models (VLMs). Image-LoRA applies low-rank adaptation only to the value path of attention layers within the visual-token span, reducing adapter-only training FLOPs roughly in proportion to the visual-token fraction. We further adapt only a subset of attention heads, selected using head influence scores estimated with a rank-1 Image-LoRA, and stabilize per-layer updates via selection-size normalization. Across screen-centric grounding and referring benchmarks spanning text-heavy to image-heavy regimes, Image-LoRA matches or closely approaches standard LoRA accuracy while using fewer trainable parameters and lower adapter-only training FLOPs. The method also preserves the pure-text reasoning performance of VLMs before and after fine-tuning, as further shown on GSM8K.

Figures (11)

• Figure 1: Image-LoRA vs. Standard LoRA. Comparison of accuracy and adapter-only training FLOPs on ScreenSpot-Pro. Points are connected across increasing input-text:image token ratios (1:2 $\rightarrow$ 1:5), where we control the ratios by dynamically adjusting image sizes. FLOPs are token-limited and computed as forward + backward multiply–adds.
• Figure 2: Overview.Left: We evaluate Image-LoRA on grounded vision–language reasoning datasets, including ScreenSpot-Pro li2025screenspot and RefCOCO kazemzadeh-etal-2014-referitgame, where the model takes a text query with its system prompt and outputs a point indicating the referred object. We further evaluate on the pure text reasoning dataset GSM8K cobbe2021gsm8k to confirm that Image-LoRA does not affect pure text reasoning, and on ViLP luo2025probing containing VQAs both aligned with and against language priors to show that Image-LoRA may mitigate shortcutting through such priors. Right: Image-LoRA is applied only to selected heads along the value path of attention layers within the visual-token span (the pink area).
• Figure 3: Top: Share one $A$ per layer across its selected heads; learn $B^{(h)}$ only for the selected heads. Bottom: For the selected heads, we update the value vectors $v_t$ of the attention layer only on the visual-token span $\mathcal{I}_v$ and not on the text-token positions.
• Figure 4: Head selection for Qwen2.5-VL-7B under a input-text:image token ratio of 1:2.
• Figure 5: Head selection patterns with different $\tau$. The head selection procedure uses default hyper-parameters $\rho = 2$. All results are obtained on ScreenSpot-Pro using the 1:2 input-text:image token ratio. Intuitively, $\tau = 0$ yields an approximately uniform allocation across layers, and $\tau = 1$ makes the budget proportional to mass$\Phi_L$, We use $\tau = 0.5$ in our main experiments. For Qwen2.5-VL-7B with $28$ layers and $K_{\text{sel}} = 28$, the $\tau = 0$ setting reduces to select exactly one KV head per layer — the one with the highest importance score.