Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting

Xu Yang, Gady Agam

TL;DR

MSLoRA addresses the efficiency problem of adapting large vision models by proposing a backbone-freezing, parameter-efficient adapter that reweights feature responses. It combines a low-rank linear projection with a multi-scale nonlinear transformation, fused via element-wise multiplication and augmented by a residual connection to modulate spatial and channel attention across CNNs and ViTs. Extensive ablations and cross-architecture experiments show MSLoRA achieves competitive or superior performance to full fine-tuning on classification, detection, and segmentation while using a small fraction of trainable parameters, enabling fast convergence and strong generalization. The work highlights a practical paradigm for scalable, interpretable adaptation of frozen vision backbones with potential for broader application in PEFT and multi-task transfer learning.

Abstract

We introduce MSLoRA, a backbone-agnostic, parameter-efficient adapter that reweights feature responses rather than re-tuning the underlying backbone. Existing low-rank adaptation methods are mostly confined to vision transformers (ViTs) and struggle to generalize across architectures. MSLoRA unifies adaptation for both convolutional neural networks (CNNs) and ViTs by combining a low-rank linear projection with a multi-scale nonlinear transformation that jointly modulates spatial and channel attention. The two components are fused through pointwise multiplication and a residual connection, yielding a lightweight module that shifts feature attention while keeping pretrained weights frozen. Extensive experiments demonstrate that MSLoRA consistently improves transfer performance on classification, detection, and segmentation tasks with roughly less than 5\% of backbone parameters. The design further enables stable optimization, fast convergence, and strong cross-architecture generalization. By reweighting rather than re-tuning, MSLoRA provides a simple and universal approach for efficient adaptation of frozen vision backbones.

MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting

TL;DR

MSLoRA addresses the efficiency problem of adapting large vision models by proposing a backbone-freezing, parameter-efficient adapter that reweights feature responses. It combines a low-rank linear projection with a multi-scale nonlinear transformation, fused via element-wise multiplication and augmented by a residual connection to modulate spatial and channel attention across CNNs and ViTs. Extensive ablations and cross-architecture experiments show MSLoRA achieves competitive or superior performance to full fine-tuning on classification, detection, and segmentation while using a small fraction of trainable parameters, enabling fast convergence and strong generalization. The work highlights a practical paradigm for scalable, interpretable adaptation of frozen vision backbones with potential for broader application in PEFT and multi-task transfer learning.

Abstract

We introduce MSLoRA, a backbone-agnostic, parameter-efficient adapter that reweights feature responses rather than re-tuning the underlying backbone. Existing low-rank adaptation methods are mostly confined to vision transformers (ViTs) and struggle to generalize across architectures. MSLoRA unifies adaptation for both convolutional neural networks (CNNs) and ViTs by combining a low-rank linear projection with a multi-scale nonlinear transformation that jointly modulates spatial and channel attention. The two components are fused through pointwise multiplication and a residual connection, yielding a lightweight module that shifts feature attention while keeping pretrained weights frozen. Extensive experiments demonstrate that MSLoRA consistently improves transfer performance on classification, detection, and segmentation tasks with roughly less than 5\% of backbone parameters. The design further enables stable optimization, fast convergence, and strong cross-architecture generalization. By reweighting rather than re-tuning, MSLoRA provides a simple and universal approach for efficient adaptation of frozen vision backbones.

Paper Structure

This paper contains 16 sections, 6 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Efficient Low-Rank Projection
  5. Multi-Scale Low-Rank Adaptation
  6. Reweighting Feature Attention
  7. Enhancing Nonlinearity
  8. Enhancement Strategies
  9. Parameter Analysis
  10. Experiments
  11. Default Settings
  12. Ablation Study
  13. Activation-level Comparison with Fine-tuning
  14. Experiments on Image Classification
  15. Experiments on Downstream Tasks
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Qualitative effects of attention reweighting.Top: A comparison between a fixed backbone block and our MSLoRA reweighting. Each MSLoRA layer uses fewer than 100K trainable parameters, whereas a fixed block typically exceeds 1M. Bottom: MSLoRA focuses filters at different spatial scales to emphasize task-relevant features.
  • Figure 2: MSLoRA overview. The module (left) combines (i) a low-rank linear projection hu2022lora with (ii) a multi-scale nonlinear transformation chansong2021impacts. The two branches are fused via point-wise multiplication to produce an attention reweighting map. (1) shows the minimal design; (2) groups the linear projection to reduce parameters; (3) increases nonlinearity with additional activations; and (4) applies lightweight enhancements for further gains. We illustrate placement within both residual bottlenecks and transformer blocks (right) for generality.
  • Figure 3: Comparison between standard fine-tuning (top) and MSLoRA (bottom). MSLoRA keeps the backbone frozen and learns lightweight reweighting for task-adaptive attention.
  • Figure 4: Comparison of full fine-tuning vs. MSLoRA across metrics.