Group and Shuffle: Efficient Structured Orthogonal Parametrization
Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva, Aibek Alanov, Alexey Naumov, Maxim Rakhuba
TL;DR
This work introduces GS-matrices, a Group-and-Shuffle structured class, to enable dense, parameter-efficient orthogonal parametrizations for fine-tuning pretrained models. By proving that orthogonality can be achieved via orthogonal blocks and carefully chosen permutations, the authors build GSOFT and Double GSOFT frameworks and extend the approach to GS Orthogonal Convolutions, achieving benefits in both NLP and vision tasks. The method demonstrates improved parameter-efficiency and competitive performance on GLUE, subject-driven diffusion generation, and 1-Lipschitz convolutional nets, while reducing the number of trainable parameters and FLOPs relative to prior BOFT approaches. Overall, GS-matrices offer a flexibly scalable path to efficient orthogonal fine-tuning with broad applicability and practical impact for large-scale pretrained models.
Abstract
The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it. We then use this parametrization to modify the orthogonal fine-tuning framework, improving parameter and computational efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task fine-tuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.