Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation and Its Application in Medical Image Segmentation

Guanghua He, Wangang Cheng, Hancan Zhu, Xiaohao Cai, Gaohang Yu

TL;DR

This paper tackles the high cost of fine-tuning large neural networks for medical image segmentation by introducing tCURLoRA, a tensor CUR decomposition-based PEFT method. By stacking pre-trained weight matrices into a 3D tensor and updating only the compressed tensor components via a tensor CUR (t-product) framework, it achieves substantial parameter and memory efficiency while maintaining or improving segmentation accuracy. The approach defines transformer-weight tensors, employs FFT-based sampling to construct CUR factors, and applies this fine-tuning to UNETR, showing superior Dice scores and reduced HD95 across multiple MRI datasets with a compact parameter budget (~$2.683$M, 2.98% of full fine-tuning). The results demonstrate the practical viability of tensor-based PEFT for resource-constrained medical imaging workflows and suggest promising extensions to broader, multimodal segmentation tasks.

Abstract

Transfer learning, by leveraging knowledge from pre-trained models, has significantly enhanced the performance of target tasks. However, as deep neural networks scale up, full fine-tuning introduces substantial computational and storage challenges in resource-constrained environments, limiting its widespread adoption. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed to reduce computational complexity and storage requirements by minimizing the number of updated parameters. While matrix decomposition-based PEFT methods, such as LoRA, show promise, they struggle to fully capture the high-dimensional structural characteristics of model weights. In contrast, high-dimensional tensors offer a more natural representation of neural network weights, allowing for a more comprehensive capture of higher-order features and multi-dimensional interactions. In this paper, we propose tCURLoRA, a novel fine-tuning method based on tensor CUR decomposition. By concatenating pre-trained weight matrices into a three-dimensional tensor and applying tensor CUR decomposition, we update only the lower-order tensor components during fine-tuning, effectively reducing computational and storage overhead. Experimental results demonstrate that tCURLoRA outperforms existing PEFT methods in medical image segmentation tasks.

tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation and Its Application in Medical Image Segmentation

TL;DR

This paper tackles the high cost of fine-tuning large neural networks for medical image segmentation by introducing tCURLoRA, a tensor CUR decomposition-based PEFT method. By stacking pre-trained weight matrices into a 3D tensor and updating only the compressed tensor components via a tensor CUR (t-product) framework, it achieves substantial parameter and memory efficiency while maintaining or improving segmentation accuracy. The approach defines transformer-weight tensors, employs FFT-based sampling to construct CUR factors, and applies this fine-tuning to UNETR, showing superior Dice scores and reduced HD95 across multiple MRI datasets with a compact parameter budget (~M, 2.98% of full fine-tuning). The results demonstrate the practical viability of tensor-based PEFT for resource-constrained medical imaging workflows and suggest promising extensions to broader, multimodal segmentation tasks.

Abstract

Transfer learning, by leveraging knowledge from pre-trained models, has significantly enhanced the performance of target tasks. However, as deep neural networks scale up, full fine-tuning introduces substantial computational and storage challenges in resource-constrained environments, limiting its widespread adoption. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed to reduce computational complexity and storage requirements by minimizing the number of updated parameters. While matrix decomposition-based PEFT methods, such as LoRA, show promise, they struggle to fully capture the high-dimensional structural characteristics of model weights. In contrast, high-dimensional tensors offer a more natural representation of neural network weights, allowing for a more comprehensive capture of higher-order features and multi-dimensional interactions. In this paper, we propose tCURLoRA, a novel fine-tuning method based on tensor CUR decomposition. By concatenating pre-trained weight matrices into a three-dimensional tensor and applying tensor CUR decomposition, we update only the lower-order tensor components during fine-tuning, effectively reducing computational and storage overhead. Experimental results demonstrate that tCURLoRA outperforms existing PEFT methods in medical image segmentation tasks.
Paper Structure (12 sections, 12 equations, 2 figures, 1 table)

This paper contains 12 sections, 12 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Overall Architecture
  4. Tensor CUR Decomposition
  5. tCURLoRA for Fine-Tuning UNETR Parameters
  6. Experiments
  7. Datasets
  8. Experimental Details
  9. Experimental Results
  10. Conclusion
  11. Acknowledgments.
  12. Disclosure of Interests.

Figures (2)

  • Figure 1: Comparison of PEFT Methods: Matrix vs. Tensor CUR Decomposition. (a) Matrix CURLoRA fine-tunes weight matrices independently. (b) tCURLoRA stacks weight matrices into a 3D tensor for fine-tuning. Blue indicates frozen parameters, orange indicates updated parameters, and * denotes the tensor product.
  • Figure 2: Qualitative Segmentation Results: 2D Slices and 3D Surfaces Across Datasets.

Theorems & Definitions (1)

  • definition thmcounterdefinition