Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning

TL;DR

The paper targets the inefficiency of fully fine-tuning foundation models and the structural weaknesses of existing PEFT methods when applied to high-dimensional parameter spaces, such as 4D convolution weights. It introduces FLoRA, a generalized low-rank tensor adaptation framework that preserves the original topological structure by learning a low-rank core tensor and mode-wise projection matrices, enabling reconstruction of updates across any dimensional space. The authors provide a Tucker-inspired formulation, instantiate FLoRA for both convolutional and linear layers, and validate its effectiveness with extensive experiments across computer vision, natural language processing, and multi-modal tasks, showing consistent gains over baselines with reduced parameter updates and competitive training costs. The results demonstrate FLoRA’s robustness, scalability, and ability to maintain structural integrity while delivering state-of-the-art performance, highlighting its practical impact for efficient fine-tuning of diverse foundation models.

Abstract

Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting all parameters becomes unfeasible. To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments. Yet, almost all of these methods focus on linear weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. Alternatively, some methods can be adapted for high-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix adaptations. However, these approaches destructs the structural integrity of the involved high-dimensional spaces. To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, designed for various dimensional parameter space. Specifically, our method asserts that changes in each dimensional parameter space are based on a low-rank core space which maintains the consistent topological structure with the original space. It then models the changes through this core space alongside corresponding weights to reconstruct alterations in the original space. It effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile models it via low-rank tensor adaptation. Extensive experiments on computer vision, natural language processing and multi-modal tasks validate the effectiveness of our method.

Figures (7)

• Figure 1: Difference between LoRA and FLoRA. LoRA employs low-rank matrix adaptation for each dimensional parameter space. However, for parameter space of convolution layer, the reshaping operation required by LoRA causes adjacent elements within the kernel to be separated in the matrix, disrupting the spatial locality inherent in the original convolutional space. Conversely, FLoRA asserts that the alternations of each dimensional parameter space has a low-rank core space with the consistent topological structure. This framework enables FLoRA to effectively preserve the structural integrity of the original parameter space, such as maintaining the spatial locality in convolutional operations.
• Figure 2: The normalized performance improvement of FLoRA over other baselines
• Figure 3: Fine-tuning with FLoRA under different rank (parameter budgets).
• Figure 4: Average of the Frobenius norm of $\Delta\mathbf{W}$ and the feature amplification factor during training.
• Figure 5: Sensitivity analysis of FLoRA w.r.t. scale $s$.