Benchmarking and Enhancing VLM for Compressed Image Understanding

TL;DR

The paper benchmarks vision-language models (VLMs) on compressed images across 11 codecs, >1M samples, and 7 tasks, revealing a substantial degradation in semantic understanding under heavy compression. It formalizes the degradation as the sum of an irreducible information gap and a reducible generalization gap, and demonstrates that only the latter can be mitigated via learning. To close this gap, it introduces a universal lightweight adaptor that conditions the VLM encoder on codec information through a learned embedding and distillation loss, yielding robust performance gains across bitrates and unseen codecs. Empirically, the adaptor achieves improvements in the 10–30% range on BD-Metrics, suggesting practical impact for bandwidth-constrained multimodal applications.

Abstract

With the rapid development of Vision-Language Models (VLMs) and the growing demand for their applications, efficient compression of the image inputs has become increasingly important. Existing VLMs predominantly digest and understand high-bitrate compressed images, while their ability to interpret low-bitrate compressed images has yet to be explored by far. In this paper, we introduce the first comprehensive benchmark to evaluate the ability of VLM against compressed images, varying existing widely used image codecs and diverse set of tasks, encompassing over one million compressed images in our benchmark. Next, we analyse the source of performance gap, by categorising the gap from a) the information loss during compression and b) generalisation failure of VLM. We visualize these gaps with concrete examples and identify that for compressed images, only the generalization gap can be mitigated. Finally, we propose a universal VLM adaptor to enhance model performance on images compressed by existing codecs. Consequently, we demonstrate that a single adaptor can improve VLM performance across images with varying codecs and bitrates by 10%-30%. We believe that our benchmark and enhancement method provide valuable insights and contribute toward bridging the gap between VLMs and compressed images.

Figures (24)

• Figure 1: Visualization of VLM performance drop due to image compression and improvement by our method, measured by BD-Metric.
• Figure 2: Comparative visualization of four image compression technique: uncompressed, traditional codec (JPEG), learning-based codec (ELIC), and generative codec (StableCodec), highlighting their impact on visual clarity and semantic preservation through targeted question-answering. All forms of compression-induced distortion affect the ability of VLMs to understand images.
• Figure 3: (a) BD-Metric values of different VLMs across different compression methods for the same tasks (SEEDBench). (b) BD-Metric values for different tasks under various compression methods based on the same VLM (Qwen-VL2.5-3B). (c) Summary of our main findings.
• Figure 4: Rate-Metric curves for all types of codecs on six tasks using Qwen-VL2.5-3B. Specifically, the GQA metric is computed from five categories of questions. MMBench represents the weighted average of six evaluation dimensions, while MME denotes the aggregate of perception-related measures. OCRBench, POPE, and SEEDBench are weighted averages across five scene types, three sampling strategies, and nine spatial dimensions, respectively.
• Figure 5: Rate–Metric curves validating the scaling law of distortion robustness are presented for the InternVL3 series models with 1B, 2B, and 8B parameters. Distortion robustness is assessed using OCRBench, POPE, and SEEDBench performance drop relative to the uncompressed results.