MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models

TL;DR

This work tackles the challenge of adapting large vision-language models with limited data by introducing a shared learnable representation space that injects representation tokens into upper encoder layers, enabling richer cross-modal interactions while preserving lower-layer general knowledge. The proposed MMRL framework decouples task-specific adaptation from generalization by jointly training representation tokens and class tokens, with a cosine-based regularization to align with frozen zero-shot features; at inference, base tasks utilize both feature types, whereas novel tasks rely on class features for stronger generalization. To further enhance efficiency and interaction, MMRL++ introduces SRRA and PRC, leveraging shared low-rank representations and inter-layer token composition to reduce parameters and improve stability. Across 15 datasets, MMRL and MMRL++ yield state-of-the-art or competitive performance in base-to-novel generalization, cross-dataset transfer, domain generalization, and few-shot learning, while offering favorable computational costs and training stability.

Abstract

Large-scale pre-trained Vision-Language Models (VLMs) have significantly advanced transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, undermining their ability to generalize to new tasks. To address this, we propose Multi-Modal Representation Learning (MMRL), which introduces a shared, learnable, modality-agnostic representation space. MMRL generates space tokens projected into both text and image encoders as representation tokens, enabling more effective cross-modal interactions. Unlike prior methods that mainly optimize class token features, MMRL inserts representation tokens into higher encoder layers--where task-specific features are more prominent--while preserving general knowledge in the lower layers. During training, both class and representation features are jointly optimized: a trainable projection layer is applied to representation tokens for task adaptation, while the projection layer for class token remains frozen to retain pre-trained knowledge. To further promote generalization, we introduce a regularization term aligning class and text features with the frozen VLM's zero-shot features. At inference, a decoupling strategy uses both class and representation features for base tasks, but only class features for novel tasks due to their stronger generalization. Building upon this, we propose MMRL++, a parameter-efficient and interaction-aware extension that significantly reduces trainable parameters and enhances intra-modal interactions--particularly across the layers of representation tokens--allowing gradient sharing and instance-specific information to propagate more effectively through the network. Extensive experiments on 15 datasets demonstrate that MMRL and MMRL++ consistently outperform state-of-the-art methods, achieving a strong balance between task-specific adaptation and generalization.

Figures (7)

• Figure 1: Comprehensive comparison of harmonic mean performance between the baseline CoCoOp, the previous state-of-the-art method MMA, and our proposed MMRL and MMRL++ across 11 diverse datasets for base-to-novel generalization. Our methods demonstrate substantial improvements, consistently outperforming existing approaches.
• Figure 2: Comprehensive comparison of different efficient multimodal transfer learning approaches (i.e., those incorporating multimodal interaction mechanisms) in terms of performance and the number of trainable parameters.
• Figure 3: Comparison of the proposed MMRL and MMRL++ frameworks with representative efficient multimodal transfer learning approaches—MaPLe from prompt learning and MMA from adapter-style learning.
• Figure 4: Overview of the MMRL++ training pipeline. 'C' denotes the class token, 'B' the BOS token, 'E' the EOS token, '$\mathcal{R}$' our shared representation space, and 'R' the representation tokens. The components subject to optimization include the representation space $\mathcal{R}$, the shared representation aligner $\mathcal{F}$, residual representation aligners ($A$, $B$), and the residual patch projection head for representation tokens, while the pre-trained CLIP model remains entirely frozen. To preserve generalization knowledge, representation tokens are integrated into both encoders starting from layer $J$.
• Figure 6: Comparison of MMRL and MMRL++ with previous state-of-the-art methods on few-shot learning across 11 datasets. Detailed results on all 11 datasets are provided in the Supplementary Material.