Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models

Yutong Wang, Haiyu Wang, Sai Qian Zhang

TL;DR

Vision-Language Models suffer from heavy KV-cache and compute demands, hindering real-time deployment. QSVD proposes a unified framework that jointly compresses QKV projections via a low-rank SVD, complemented by cross-layer rank allocation and post-training quantization to produce highly efficient VLMs. Empirical results across multiple models and benchmarks show QSVD achieves accuracy improvements over prior SVD and quantization approaches while drastically reducing weight, KV-cache size, and FLOPs, with notable latency gains on GPUs. The work demonstrates practical impact for resource-constrained deployment and highlights implications for safety, robustness, and broader accessibility of multimodal transformers.

Abstract

Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability. In this work, we propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, we extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than $10\%$ accuracy improvement while consuming less hardware cost, making it better for real-time deployment on resource-constrained devices. We open source our code at \href{https://github.com/SAI-Lab-NYU/QSVD}{\texttt{https://github.com/SAI-Lab-NYU/QSVD}}.

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models

TL;DR

Vision-Language Models suffer from heavy KV-cache and compute demands, hindering real-time deployment. QSVD proposes a unified framework that jointly compresses QKV projections via a low-rank SVD, complemented by cross-layer rank allocation and post-training quantization to produce highly efficient VLMs. Empirical results across multiple models and benchmarks show QSVD achieves accuracy improvements over prior SVD and quantization approaches while drastically reducing weight, KV-cache size, and FLOPs, with notable latency gains on GPUs. The work demonstrates practical impact for resource-constrained deployment and highlights implications for safety, robustness, and broader accessibility of multimodal transformers.

Abstract

Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability. In this work, we propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, we extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than accuracy improvement while consuming less hardware cost, making it better for real-time deployment on resource-constrained devices. We open source our code at \href{https://github.com/SAI-Lab-NYU/QSVD}{\texttt{https://github.com/SAI-Lab-NYU/QSVD}}.
Paper Structure (24 sections, 29 equations, 4 figures, 9 tables)

This paper contains 24 sections, 29 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Vision Language Model
  4. Singular Value Decomposition for Large Models
  5. Quantization for Large Models
  6. Methodology
  7. Singular-Value Decomposition over Joint QKV Weights
  8. Cross-layer Rank Allocation for Low-rank SVD
  9. Post-Training Quantization Scheme for Low-Rank VLMs
  10. Evaluation Results
  11. Accuracy Evaluation on QSVD-noQ
  12. Accuracy Evaluation of QSVD
  13. Ablation Study
  14. Effectiveness of Cross-layer Rank Allocation Scheme
  15. Impact of Learning $\beta$
  16. ...and 9 more sections

Figures (4)

  • Figure 1: (a) An example on vision-language model. (b) An overview of QSVD.
  • Figure 2: Efficiency analysis of different SVD schemes. (a)(b) are original Q/K/V matrix applied SVD. (c)(d)(e)(f) are proposed concatenated QKV SVD and their corresponding decoding process.
  • Figure 3: Input activation distribution within VLM. Only partial channel are shown.
  • Figure 4: Normalized speedup of QSVD-noQ and QSVD W8A8 on low-end GPU.