HyDRA: Hierarchical and Dynamic Rank Adaptation for Mobile Vision Language Model

TL;DR

HyDRA addresses the efficiency gap in fine-tuning mobile vision-language models by introducing hierarchical and dynamic rank adaptation for Low-Rank Adaptation (LoRA). It identifies uneven layer sensitivities via average gradient norms and develops a three-phase, end-to-end optimization that assigns both coarse- and fine-grained ranks per layer/component, guided by a lightweight performance model. Empirical results on MobileLLaMA variants show HyDRA consistently surpassing LoRA across six benchmarks and, in some cases, beating full-parameter fine-tuning, all without increasing trainable parameters. This approach enables significant accuracy gains for mobile VLMs with constrained resources and has potential to generalize to other multimodal tasks, including video-language models.

Abstract

Vision Language Models (VLMs) have undergone significant advancements, particularly with the emergence of mobile-oriented VLMs, which offer a wide range of application scenarios. However, the substantial computational requirements for training these models present a significant obstacle to their practical application. To address this issue, Low-Rank Adaptation (LoRA) has been proposed. Nevertheless, the standard LoRA with a fixed rank lacks sufficient capability for training mobile VLMs that process both text and image modalities. In this work, we introduce HyDRA, a parameter-efficient fine-tuning framework designed to implement hierarchical and dynamic rank scheduling for mobile VLMs. This framework incorporates two essential optimization strategies: (1) hierarchical optimization, which involves a coarse-grained approach that assigns different ranks to various layers, as well as a fine-grained method that adjusts ranks within individual layers, and (2) dynamic adjustment, which employs an end-to-end automatic optimization using a lightweight performance model to determine and adjust ranks during the fine-tuning process. Comprehensive experiments conducted on popular benchmarks demonstrate that HyDRA consistently outperforms the baseline, achieving a 4.7\% improvement across various model sizes without increasing the number of trainable parameters. In some tasks, it even surpasses full-parameter fine-tuning.

Figures (6)

• Figure 1: An illustration of hierarchical and dynamic rank adaptation. The average gradient norms serve as the basis for assigning the rank of layers. A lightweight performance model determines the optimal set of rank values.
• Figure 2: Overview of HyDRA with hierarchical rank optimization and dynamic adjustment for mobile VLMs instruction tuning. $X_t$ and $X_v$ represent the text and image tokens, respectively. $R^{Up}$ represents the rank for the up projection of the feed- forward neural network (FFN). $R^Q$, $R^K$, $R^{Gate}$, etc., are similarly defined.
• Figure 3: Average gradient norms of each layer with LoRA.
• Figure 4: Stage partitioning in the computation graph.
• Figure 5: An end-to-end learning-based framework for optimizing hierarchical and dynamic ranks in mobile VLMs instruction tuning. Initialization Phase: Define the solution space using average gradient norms. Rank Assignment Phase: Assign ranks and conducts instruction tuning accordingly. Iterative Enhancement Phase: Evaluate mobile VLMs on downstream tasks to obtain $p(Z)$, then iteratively refine the performance model $\Phi$ and predict the optimal rank scheduling $Z^*$.