Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration

TL;DR

The paper tackles the quadratic computational burden of multimodal context in MLLMs by introducing FiCoCo, a training-free framework that trims visual tokens during prefilling while recovering critical information. FiCoCo-V operates in the vision encoder to prune redundancy via a redundancy-based discard, correlation-based recycling, and self-preserving compression; FiCoCo-L extends this idea into the LLM decoder using task-aware textual priors. Across image and video benchmarks, FiCoCo achieves up to 14.7x FLOPs reduction with up to ~94% retention of performance, outperforming prior training-free baselines and generalizing across architectures and tasks. The work provides thorough theoretical FLOPs analysis, ablations, and extensive experiments, demonstrating practical acceleration without retraining and broad applicability.

Abstract

The quadratic complexity of Multimodal Large Language Models (MLLMs) with respect to context length poses significant computational and memory challenges, hindering their real-world deployment. In the paper, we devise a ''filter-correlate-compress'' framework to accelerate the MLLM by systematically optimizing multimodal context length during prefilling. The framework first implements FiCoCo-V, a training-free method operating within the vision encoder. It employs a redundancy-based token discard mechanism that uses a novel integrated metric to accurately filter out redundant visual tokens. To mitigate information loss, the framework introduces a correlation-based information recycling mechanism that allows preserved tokens to selectively recycle information from correlated discarded tokens with a self-preserving compression, thereby preventing the dilution of their own core content. The framework's FiCoCo-L variant further leverages task-aware textual priors to perform token reduction directly within the LLM decoder. Extensive experiments demonstrate that the FiCoCo series effectively accelerates a range of MLLMs, achieves up to 14.7x FLOPs reduction with 93.6% performance retention. Our methods consistently outperform state-of-the-art training-free approaches, showcasing effectiveness and generalizability across model architectures, sizes, and tasks without requiring retraining. Code: https://github.com/kawhiiiileo/FiCoCo

Figures (10)

• Figure 1: The comparison to existing token reduction methods. Our FiCoCo series achieves state-of-the-art results with five popular MLLMs across benchmarks.
• Figure 2: Overview of FiCoCo-V and FiCoCo-L. Due to the two methods being applied to different modules (vision encoder and LLM decoder), they have different implementations for summarized redundancy and correlation matrix in the filter and correlate stages. Simultaneously, the compression modules of FiCoCo-V and FiCoCo-L are identical, both employing self-preserving compression based on the correlation matrix.
• Figure 3: Layer-wise distribution of Pairwise Token Correlation Gap. We observe that the difference between the top-1 and top-2 correlation scores for each discarded token displays a high variance, highlighting the inadequacy of a fixed number of correlated preserved tokens.
• Figure 4: Visualizations of token discard and information recycling by (a) FiCoCo-V and (b) FiCoCo-L. Red: traced patch token; Green: recycling destination.
• Figure 5: Heatmap visualization using different inputs as the mean reference.