Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs

TL;DR

TT-LoRA introduces a tensor-train based parameter-efficient fine-tuning approach that forgoes adapters and directly decomposes the weight-update matrix, freezing the pre-trained weights to achieve dramatic parameter reductions. By representing the update as a TT decomposition with cores $\mathcal{C}_i$ and controlling capacity via TT ranks, TT-LoRA achieves orders-of-magnitude compression while maintaining or exceeding the accuracy of competing PEFT methods on both BERT-family and LLaMA-family tasks. Extensive experiments on GLUE and SuperGLUE demonstrate superior or comparable performance with dramatically fewer trainable parameters, along with memory-efficient storage suitable for resource-constrained devices. The work provides a thorough hyperparameter study using HyperBand to map the compression-performance trade-offs and outlines a path toward scaling TT-LoRA to even larger models, potentially broadening access to high-performance LLM fine-tuning.

Abstract

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing (NLP) tasks, such as question-answering, sentiment analysis, text summarization, and machine translation. However, the ever-growing complexity of LLMs demands immense computational resources, hindering the broader research and application of these models. To address this, various parameter-efficient fine-tuning strategies, such as Low-Rank Approximation (LoRA) and Adapters, have been developed. Despite their potential, these methods often face limitations in compressibility. Specifically, LoRA struggles to scale effectively with the increasing number of trainable parameters in modern large scale LLMs. Additionally, Low-Rank Economic Tensor-Train Adaptation (LoRETTA), which utilizes tensor train decomposition, has not yet achieved the level of compression necessary for fine-tuning very large scale models with limited resources. This paper introduces Tensor Train Low-Rank Approximation (TT-LoRA), a novel parameter-efficient fine-tuning (PEFT) approach that extends LoRETTA with optimized tensor train (TT) decomposition integration. By eliminating Adapters and traditional LoRA-based structures, TT-LoRA achieves greater model compression without compromising downstream task performance, along with reduced inference latency and computational overhead. We conduct an exhaustive parameter search to establish benchmarks that highlight the trade-off between model compression and performance. Our results demonstrate significant compression of LLMs while maintaining comparable performance to larger models, facilitating their deployment on resource-constraint platforms.

Figures (5)

• Figure 1: Performance vs. trainable parameters comparison between TT-LoRA and various PEFT methods on DeBERTa model using the GLUE benchmark
• Figure 2: Various Parameter-efficient Fine-Tuning Approaches
• Figure 3: Overview of the proposed TT-Lora framework: (a) BERT Architecture, (b) Encoder layer of BERT Architecture, (c) TT-LoRA wrapping the value vector of BERT encoder: the pre-trained weight, $W_0$, is kept frozen while the update to the pre-trained weight, $\Delta W$, is decomposed into small tensors.
• Figure 4: Memory storage comparison for trainable parameters of various PEFT methods and LLMs.
• Figure 5: Model compression vs. accuracy of LLaMA2-7B model with TT-LoRA having different tensor format and other PEFT approaches on SuperGLUE BoolQ benchmark.