When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning

TL;DR

The paper tackles the instability of rank-1 parameter-efficient fine-tuning in vision–language models by revealing that optimization is dominated by a translation-like cross-modal gap in activation space. It introduces Gap-Init, a geometry-aware initialization that aligns the rank-1 update with an empirically estimated modality-gap vector from a small calibration set, while keeping the initial update at zero. Across COCO, VQA, and zero-shot transfers, Gap-Init stabilizes rank-1 training and often matches or exceeds rank-8 baselines with substantially fewer trainable parameters. The work demonstrates that in extreme low-rank regimes, the alignment of the update direction with the underlying geometry can be as important as the rank itself, highlighting the central role of activation-space geometry in PEFT for multimodal models.

Abstract

Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings -- especially rank-1 LoRA -- are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.

Figures (5)

• Figure 1: Gap-Init stabilizes rank-1 adaptation via geometry-aware initialization across multimodal tasks and training settings.(a) Image Captioning (COCO). Rank-1 Gap-Init matches or slightly exceeds strong rank-8 baselines in this setting while using substantially fewer trainable parameters. (b) Visual Question Answering (VQAv2). Gap-Init improves rank-1 accuracy over random initialization and yields stable convergence. (c) Geometric intuition. Vision and text embeddings form anisotropic cones; their mismatch admits a translation-like component that is particularly consequential for optimization at rank 1. Random rank-1 initialization typically has negligible projection onto this axis, suppressing useful gradient flow. Gap-Init aligns the update direction with the estimated gap vector, improving optimization dynamics from the first step and throughout training across tasks.
• Figure 2: Near-orthogonality at initialization. Random rank-1 LoRA directions concentrate near $\cos(u,b)\approx 0$, whereas Gap-Init aligns the direction with the estimated gap axis by construction.
• Figure 3: Spectrum of instance-specific gap vectors. The singular-value spectrum suggests a prominent leading direction, with diminishing contributions from higher-rank components.
• Figure 4: Intrinsic geometry (cone effect). After centering, both vision and text embeddings remain strongly concentrated around their mean directions, indicating anisotropy within each modality.
• Figure 5: Gradient suppression in high dimensions (Gaussian toy model). We compute the effective gradient magnitude $|\langle b, \hat{g} \rangle|$ for Rank-1 LoRA directions in dimensions $d \in [64, 4096]$. Random initialization exhibits the predicted $\mathcal{O}(d^{-1/2})$ decay (red), showing that almost all gradient signal along the modality gap vanishes at high dimensionality. In contrast, Gap-Init (green) maintains $\mathcal{O}(1)$ signal independent of dimensionality. This verifies that optimization collapse in Rank-1 LoRA is a consequence of high-dimensional geometry, not insufficient rank.

Theorems & Definitions (2)

• Proposition 3.1: Concentration of random directions
• Theorem 1.1: Optimization Collapse