LSR-Adapt: Ultra-Efficient Parameter Tuning with Matrix Low Separation Rank Kernel Adaptation
Xin Li, Anand Sarwate
TL;DR
This work addresses the challenge of parameter-efficient fine-tuning for large models by introducing the Low Separation Rank (LSR) kernel, which kernelizes low-rank adapters using matrix separated representations. By expressing the weight update as a product of two separable Kronecker-structured factors, the approach dramatically reduces trainable parameters while maintaining or improving accuracy, and it aligns well with GPU parallelism. The method is validated on GLUE and SuperGLUE with RoBERTa, showing competitive performance at roughly a quarter of the LoRA parameter count in some configurations. The contributions include a principled theoretical foundation for the representation, practical design for PEFT, and implications for high-performance GPU implementations, with future work focusing on optimized Kronecker-based kernels.
Abstract
Imposing an effective structural assumption on neural network weight matrices has been the major paradigm for designing Parameter-Efficient Fine-Tuning (PEFT) systems for adapting modern large pre-trained models to various downstream tasks. However, low rank based adaptation has become increasingly challenging due to the sheer scale of modern large language models. In this paper, we propose an effective kernelization to further reduce the number of parameters required for adaptation tasks. Specifically, from the classical idea in numerical analysis regarding matrix Low-Separation-Rank (LSR) representations, we develop a kernel using this representation for the low rank adapter matrices of the linear layers from large networks, named the Low Separation Rank Adaptation (LSR-Adapt) kernel. With the ultra-efficient kernel representation of the low rank adapter matrices, we manage to achieve state-of-the-art performance with even higher accuracy with almost half the number of parameters as compared to conventional low rank based methods. This structural assumption also opens the door to further GPU-side optimizations due to the highly parallelizable nature of Kronecker computations.