LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

TL;DR

LoRETTA introduces ultra-low-parameter fine-tuning for large language models by leveraging tensor-train decomposition to shrink trainable parameters. It presents two variants, LoRETTA_adp and LoRETTA_rep, combining tensorized adapters and TT-based reparameterization to achieve up to 100x parameter savings while maintaining or improving performance on GLUE-scale tasks and large-scale LLaMA-2 models. The approach yields tangible benefits in training efficiency, memory footprint, and robustness to overfitting, with strong multi-task learning performance across NLP tasks and generation settings. Extensive rank-ablation analyses and memory/performance benchmarks demonstrate the practicality of TT-based fine-tuning for resource-constrained environments, and code release plans support broad adoption and reproducibility.

Abstract

Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. However, existing PEFT methods are still limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). To address this challenge, we present LoRETTA, an ultra-parameter-efficient framework that significantly reduces trainable parameters through tensor-train decomposition. Specifically, we propose two methods, named {LoRETTA}$_{adp}$ and {LoRETTA}$_{rep}$. The former employs tensorized adapters, offering a high-performance yet lightweight approach for the fine-tuning of LLMs. The latter emphasizes fine-tuning via weight parameterization with a set of small tensor factors. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to $100\times$ fewer parameters on the LLaMA-2-7B models. Furthermore, empirical results demonstrate that the proposed method effectively improves training efficiency, enjoys better multi-task learning performance, and enhances the anti-overfitting capability. Plug-and-play codes built upon the Huggingface framework and PEFT library will be released.

Figures (5)

• Figure 1: The performance vs. trainable parameters on the DeBERTa-Base, showcasing the relationship between parameter efficiency and performance across various GLUE tasks.
• Figure 2: Architecture of LoRETTA$_{adp}$ for the transformer encoders or decoders. $*$ the tensorized classifier is optional for different tasks. For classification tasks, we set this part to be trainable and we freeze this part during language modeling tasks.
• Figure 3: Architecture of the LoRETTA$_{rep}$ method for a single transformer encoder.
• Figure 4: Evaluation loss comparison across various PEFT methods on the DeBERTa-base model. The loss is smoothed with a window size of 20 and the shallow means the standard deviation boundaries.
• Figure 5: Comparison of memory storage for trainable parameters across different models and methods.