LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression

TL;DR

This work addresses the memory constraints of multi-modal LVLMs by introducing LVLM-Compress-Bench, a plug-and-play framework to benchmark the effects of dynamic KV cache and static weight compression on LVLM generation and societal trust metrics. It combines activation-aware weight quantization (AWQ) with a spectrum of KV cache quantization schemes (Uniform, Outlier-reduced, and Group-wise, including hybrids like g-KCVT) across multiple LLaVA variants and ten diverse benchmarks. The study uncovers that carefully designed KV quantization (notably 2-bit KV with g-KC$_{128}$VT$_{128}$) can preserve accuracy close to FP16, while joint KV+weight quantization can achieve memory savings and sometimes even improve performance on certain tasks; larger models generally tolerate lower precision better, though weight precision remains a sensitivity factor for smaller models. These findings offer practical guidance for deploying memory-efficient LVLMs on edge devices and in privacy-conscious scenarios, while highlighting avenues for future work in pruning and low-rank techniques to expand the compression toolkit with robust ethical safeguards.

Abstract

Despite recent efforts in understanding the compression impact on large language models (LLMs) in terms of their downstream task performance and trustworthiness on relatively simpler uni-modal benchmarks (for example, question answering, common sense reasoning), their detailed study on multi-modal Large Vision-Language Models (LVLMs) is yet to be unveiled. Towards mitigating this gap, we present LVLM-Compress-Bench, a framework to first thoroughly study the broad impact of compression on the generative performance of LVLMs with multi-modal input driven tasks. In specific, we consider two major classes of compression for autoregressive models, namely KV cache and weight compression, for the dynamically growing intermediate cache and static weights, respectively. We use four LVLM variants of the popular LLaVA framework to present our analysis via integrating various state-of-the-art KV and weight compression methods including uniform, outlier-reduced, and group quantization for the KV cache and weights. With this framework we demonstrate on ten different multi-modal datasets with different capabilities including recognition, knowledge, language generation, spatial awareness, visual reasoning, hallucination and visual illusion identification, toxicity, stereotypes and bias. In specific, our framework demonstrates the compression impact on both general and ethically critical metrics leveraging a combination of real world and synthetic datasets to encompass diverse societal intersectional attributes. Extensive experimental evaluations yield diverse and intriguing observations on the behavior of LVLMs at different quantization budget of KV and weights, in both maintaining and losing performance as compared to the baseline model with FP16 data format. Code will be open-sourced at https://github.com/opengear-project/LVLM-compress-bench.

Figures (6)

• Figure 1: Details of LVLM-Compress-Bench framework. We use this framework to benchmark with respect to the uncompressed baseline model with FP16 format. Notably, we consider different "plug-and-play" compression in the framework where the compressed model does not need any post-compression fine-tuning. This framework identifies the performance and societal trust impact of both the uncompressed and compressed model variants.
• Figure 2: (a) Gender-occupation bias (male-female), (b) race-crime (white-black), and (c) race-status (white-black) association scores evaluated on PAIRS.
• Figure 3: (a) Mean of max toxicity (lower better), (b) competence (higher better), and (c) stereotype (lower better) scores on a physical-gender subset (5K) of SocialCounterFactuals dataset, evaluated for the 'Keywords' prompt.
• Figure 4: Toxicity, competence, and stereotype scores with a subset of SocialCounterFactuals dataset when evaluated for the 'Keywords' prompt, with both KV and weight quantization (W$n$KV$m$), with $n$ and $m$ being the quantization bit-wdith for W and KV, respectively.
• Figure 5: Ablation with different group quantization, (a) different bit-width for K and V cache for different grouping, here k$n$v$m$ means K and V cache in $n$ and $m$-bit, respectively. (b) different forms of hybrid grouping. We use MM-Vet.