ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task

TL;DR

ToFu introduces a training-free visual token fusion that reduces the prefix token load for large multimodal models without retraining or architecture changes. It sequentially merges similar visual tokens according to a dynamic cosine-similarity threshold, yielding a compact token set that preserves distinctive information. Evaluations on LLaVA-Interleave and the newly proposed ComPairs benchmark show substantial token and memory reductions (around 60% and up to ~66%, respectively) with minimal or even positive impacts on accuracy for many backbones, particularly in challenging multi-image tasks. The approach is encoder-agnostic and readily applicable to existing LMM pipelines, offering practical efficiency gains for high-resolution, multi-image reasoning scenarios.

Abstract

Large Multimodal Models (LMMs) are powerful tools that are capable of reasoning and understanding multimodal information beyond text and language. Despite their entrenched impact, the development of LMMs is hindered by the higher computational requirements compared to their unimodal counterparts. One of the main causes of this is the large amount of tokens needed to encode the visual input, which is especially evident for multi-image multimodal tasks. Recent approaches to reduce visual tokens depend on the visual encoder architecture, require fine-tuning the LLM to maintain the performance, and only consider single-image scenarios. To address these limitations, we propose ToFu, a visual encoder-agnostic, training-free Token Fusion strategy that combines redundant visual tokens of LMMs for high-resolution, multi-image, tasks. The core intuition behind our method is straightforward yet effective: preserve distinctive tokens while combining similar ones. We achieve this by sequentially examining visual tokens and deciding whether to merge them with others or keep them as separate entities. We validate our approach on the well-established LLaVA-Interleave Bench, which covers challenging multi-image tasks. In addition, we push to the extreme our method by testing it on a newly-created benchmark, ComPairs, focused on multi-image comparisons where a larger amount of images and visual tokens are inputted to the LMMs. Our extensive analysis, considering several LMM architectures, demonstrates the benefits of our approach both in terms of efficiency and performance gain.

Figures (3)

• Figure 1: Our proposed ToFu allows reducing the number of visual tokens in input to the LLM in LMMs while preserving the same information. This allows reducing runtime and GPU memory consmption without affecting the performance. The image refers to the InternVL2 4B model and the GPU NVIDIA L40S with 48GB of memory.
• Figure 2: Overall accuracy (y-axis) and percentage of tokens (x-axis) used by InternVL2-4B on the LLava-Interleave Benchmark. We compare our ToFu, which has a dynamic threshold (red line), against a variant of ToFu with fixed $\tau$ (blue line) that applies the same threshold $\tau$ for each image in the benchmark and the vanilla InternVL2-4B which uses all tokens (green line). For each value of the threshold, we indicate the corresponding average number of tokens and the standard deviation as a semi-transparent circle.
• Figure 3: Sample from the ComPairs dataset where we show the benefits and the results obtained with the ToFu algorithm with respect to the vanilla approach. For space reasons, we have abbreviated the word "Product" in the image to "Prod."