Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models

TL;DR

This work reconceptualizes fine-tuning of Vision-Language Models as a structural reparameterization problem, introducing Mask Fine-Tuning (MFT) which freezes the backbone and learns masks to reconfigure internal subnetworks. The authors develop both Hard (H-MFT) and Soft (S-MFT) mask variants, deriving mask parameterizations, training objectives, and a PAC-Bayes grounded theoretical perspective. Empirically, S-MFT consistently surpasses strong PEFT baselines like LoRA and can exceed full fine-tuning performance while keeping weights fixed, across multiple language backbones and vision towers. The approach demonstrates robust scalability, efficiency, and interpretable mask patterns, suggesting a practical, plug-and-play alternative for adapting large multimodal models to downstream tasks.

Abstract

Explorations in fine-tuning Vision-Language Models (VLMs), such as Low-Rank Adaptation (LoRA) from Parameter Efficient Fine-Tuning (PEFT), have made impressive progress. However, most approaches rely on explicit weight updates, overlooking the extensive representational structures already encoded in pre-trained models that remain underutilized. Recent works have demonstrated that Mask Fine-Tuning (MFT) can be a powerful and efficient post-training paradigm for language models. Instead of updating weights, MFT assigns learnable gating scores to each weight, allowing the model to reorganize its internal subnetworks for downstream task adaptation. In this paper, we rethink fine-tuning for VLMs from a structural reparameterization perspective grounded in MFT. We apply MFT to the language and projector components of VLMs with different language backbones and compare against strong PEFT baselines. Experiments show that MFT consistently surpasses LoRA variants and even full fine-tuning, achieving high performance without altering the frozen backbone. Our findings reveal that effective adaptation can emerge not only from updating weights but also from reestablishing connections among the model's existing knowledge. Code available at: https://github.com/Ming-K9/MFT-VLM

Figures (9)

• Figure 1: Overall performance comparison of different fine-tuning methods on VLM with Qwen2.5-0.5B as the language model. Our S-MFT consistently surpasses both FFT and LoRA-based approaches on all tasks. Among the variants, S-MFT Both achieves the strongest overall performance, highlighting the effectiveness of jointly masking both Attention and MLP layers.
• Figure 2: Overview of Mask Fine-Tuning (MFT) applied to a Vision-Language Model (VLM). The pre-trained parameters are all frozen during the training, while learnable scores are added to pre-trained weights in the projector and language model. We have two strategies to generate either hard or soft masks that modulate the effective weights.
• Figure 3: Ablation study on the initialization value of the score matrix and temperature across four language backbones. Each surface plot illustrates the performance variation of MMMU benchmark of Soft Mask Fine-Tuning (S-MFT) under different hyperparameter combinations. The red dashed line in each plot marks the optimal configuration for the corresponding backbone.
• Figure 4: Performance comparison of S-MFT, FFT, and LoRA under varying training data proportions across four VLM backbones. Each curve shows the performance with standard deviation of the MMMU benchmark as the proportion of training data increases. S-MFT consistently maintains strong performance across most data proportions, showing robustness regardless of data scale.
• Figure 5: Layer-wise analysis of S-MFT on Qwen2.5-0.5B and TinyLLaMA-1.1B. Each curve shows the average performance when masks are applied only to a subset of layers, compared with full-layer training (red dashed line).