Low-Rank Approximation, Adaptation, and Other Tales
Jun Lu
TL;DR
This survey analyzes how low-rank representations can be learned and exploited in data analysis and large language model tuning. It presents alternating least squares as a core method for obtaining low rank approximations and extends it to Hadamard, Kronecker, and Khatri-Rao decompositions, including missing data scenarios. It then connects these decompositions to practical model adaptation techniques for transformers, introducing LoRA, LoHA, LoKr, and LoKH as efficient fine tuning strategies with theoretical and empirical implications for memory, latency, and flexibility. The work highlights the algebraic structure of special matrix products to enable closed-form updates or gradient-based optimization, and demonstrates how these ideas can compress and adapt large neural networks while maintaining performance. Overall, the paper provides both foundational algorithms and forward-looking adaptation strategies for scalable, structure-preserving model customization.
Abstract
Low-rank approximation is a fundamental technique in modern data analysis, widely utilized across various fields such as signal processing, machine learning, and natural language processing. Despite its ubiquity, the mechanics of low-rank approximation and its application in adaptation can sometimes be obscure, leaving practitioners and researchers with questions about its true capabilities and limitations. This paper seeks to clarify low-rank approximation and adaptation by offering a comprehensive guide that reveals their inner workings and explains their utility in a clear and accessible way. Our focus here is to develop a solid intuition for how low-rank approximation and adaptation operate, and why they are so effective. We begin with basic concepts and gradually build up to the mathematical underpinnings, ensuring that readers of all backgrounds can gain a deeper understanding of low-rank approximation and adaptation. We strive to strike a balance between informal explanations and rigorous mathematics, ensuring that both newcomers and experienced experts can benefit from this survey. Additionally, we introduce new low-rank decomposition and adaptation algorithms that have not yet been explored in the field, hoping that future researchers will investigate their potential applicability.