Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

TL;DR

This work reveals a task-setup mismatch in evaluating visual token compression for multimodal LLMs, showing that many benchmarks are biased toward easy samples and that simple image downsampling often outperforms advanced compression methods. It conducts extensive experiments comparing token compression techniques against downsampling across multiple benchmarks, identifying a simplicity bias and proposing a data-filtering approach. The authors introduce VTC-Bench, a three-step framework that uses downsampling to label samples by difficulty and builds compression-focused benchmark subsets with clear upper and lower bounds for evaluation. By denoising benchmarks and focusing on truly compression-relevant samples, the framework aims to foster fairer assessments and more meaningful progress in visual token compression for efficient MLLMs.

Abstract

Recent endeavors to accelerate inference in Multimodal Large Language Models (MLLMs) have primarily focused on visual token compression. The effectiveness of these methods is typically assessed by measuring the accuracy drop on established benchmarks, comparing model performance before and after compression. However, these benchmarks are originally designed to assess the perception and reasoning capabilities of MLLMs, rather than to evaluate compression techniques. As a result, directly applying them to visual token compression introduces a task mismatch. Strikingly, our investigation reveals that simple image downsampling consistently outperforms many advanced compression methods across multiple widely used benchmarks. Through extensive experiments, we make the following observations: (i) Current benchmarks are noisy for the visual token compression task. (ii) Down-sampling is able to serve as a data filter to evaluate the difficulty of samples in the visual token compression task. Motivated by these findings, we introduce VTC-Bench, an evaluation framework that incorporates a data filtering mechanism to denoise existing benchmarks, thereby enabling fairer and more accurate assessment of visual token compression methods. All data and code are available at https://github.com/Chenfei-Liao/VTC-Bench.

Figures (4)

• Figure 1: (a) Average Decline Ratio (ADR) of five visual token compression methods on eight benchmarks (Model: Qwen2-VL-7B; Benchmark: as shown in Table \ref{['table1']}; Device: 1 A800)). (b) Inference time per image comparison of DART and Downsample (Model: Qwen2-VL-7B; Benchmark: MMstar; Compression Ratio: 0.75; Device: 1 A800).
• Figure 2: Comparison of advanced token compression methods and downsampling on Qwen2-VL-7B by groups at 75% compression.
• Figure 3: The VTC-Bench is a simple but effective framework that can transform any existing benchmarks to a subset that can fairly evaluate VTC (Visual Token Compression) methods. The samples that are answered correctly by the original Qwen2-VL model without downsampling form the input samples. More details in Sec. \ref{['Framwork']}.
• Figure 4: VTC-Bench results on Qwen2-VL-7B.