Low-Rank Tensor Approximation of Weights in Large Language Models via Cosine Lanczos Bidiagonalization
A. El Ichi, K. Jbilou
TL;DR
The paper tackles the large memory and compute demands of Transformer-based LLMs by introducing a tensor-based compression framework using the c-product within the \\mathcal{L}-product. It develops TLASER, a Tensor Layer-Selective Rank Reduction method that tensorizes Transformer weights to expose multi-head and block structure, and uses tensor Lanczos bidiagonalization to efficiently extract dominant spectral components. By applying per-layer, energy-aware, and structure-preserving low-rank approximations, TLASER achieves substantial compression without retraining, preserving important cross-head and intra-head correlations. Experimental results on GPT-J-6B with TruthfulQA demonstrate favorable accuracy and loss behavior for targeted, single-layer interventions, highlighting TLASER's potential for scalable deployment of large language models.
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language tasks but suffer from extremely large memory footprints and computational costs. In this paper, we introduce a tensor compression framework based on the cproduct for computing low rank approximation In the first part of our approach, we leverage the algebraic structure of the cproduct to represent weight tensors such as those in embedding layers, attention projections, and feed forward networks in a transform domain where frontal slices can be jointly approximated by low rank tensor factors. This enables computationally efficient compression that exploits multidimensional correlations beyond traditional SVD methods.