Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

TL;DR

<3-5 sentence high-level summary> The paper shows that tuning only LayerNorm in attention blocks suffices to adapt LLMs to multi-modal tasks, offering dramatic parameter and memory efficiency while achieving competitive or superior performance versus full finetuning and LoRA. Framed as domain adaptation, LayerNorm tuning demonstrates improved expressive power and reduced gradient variance; adding conversational data further boosts efficiency. The work includes extensive experiments on 7B/13B models across MME, VQA, COCO, Flickr30k, and POPE benchmarks, with analysis of vision-language connectors and data types. It provides a practical pathway for scalable, resource-efficient fine-tuning of multimodal LLMs on consumer hardware and guides future PEFT method development.

Abstract

This paper introduces an efficient strategy to transform Large Language Models (LLMs) into Multi-Modal Large Language Models (MLLMs). By conceptualizing this transformation as a domain adaptation process, i.e., transitioning from text understanding to embracing multiple modalities, we intriguingly note that, within each attention block, tuning LayerNorm suffices to yield strong performance. Moreover, when benchmarked against other tuning approaches like full parameter finetuning or LoRA, its benefits on efficiency are substantial. For example, when compared to LoRA on a 13B model scale, performance can be enhanced by an average of over 20% across five multi-modal tasks, and meanwhile, results in a significant reduction of trainable parameters by 41.9% and a decrease in GPU memory usage by 17.6%. On top of this LayerNorm strategy, we showcase that selectively tuning only with conversational data can improve efficiency further. Beyond these empirical outcomes, we provide a comprehensive analysis to explore the role of LayerNorm in adapting LLMs to the multi-modal domain and improving the expressive power of the model.

Figures (8)

• Figure 1: (left) Different tuning methods for MLLMs. Trainable components are in blue, while frozen parameters are in gray. Within the attention blocks, (a) only activates LayerNorm parameters. Note that vision-language connector, word embedding, and output head paramters are by default activated for all three options. (right) Comparison on trainable parameters and GPU memory. Tuning LayerNorm achieves significant reductions in trainable parameters and GPU memory usages.
• Figure 2: Performances of models that are finetuned on different datasets on four multi-modal benchmarks. The MME score is the sum of both Cognition and Perception scores on the benchmark.
• Figure 3: Layer similarities between different LLM layers in (a) Finetuned and (b) LayerNorm-tuned MM-Vicuna-7B. The average layer similarity of two models are 0.624 and 0.585, respectively.
• Figure 4: Gradients of the input LayerNorm in the 11th layer of the MM-Vicuna as training proceeds. LayerNorm-tuned model has lower gradient variance than full parameter finetuning.
• Figure A1: Layer similarities between different LLM layers in (a) Finetuned and (b) LayerNorm-tuned MM-LLaMA2-7B.