D$^{3}$ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs

TL;DR

Diffusion-based multimodal LLMs suffer from cubic decoding complexity due to full self-attention over thousands of visual tokens across $T= ext{O}(N)$ denoising steps. D3ToM introduces a decider-guided dynamic token merging mechanism that, at each step, uses decider tokens to build a visual-token importance map and merges the non-salient tokens into their most similar kept ones, effectively shortening the sequence without changing model parameters and with a timestep-aware merge schedule. The approach is training-free, plug-and-play, and compatible with KV-Cache, delivering strong accuracy retention (roughly 96%+ at extreme reductions) while achieving up to ~70% FLOPs savings and ~2.3x faster inference across seven multimodal benchmarks. This work significantly reduces the practical cost of diffusion MLLMs and broadens their accessibility, while remaining compatible with existing efficiency tricks and enabling future extensions to other backbones and encoders.

Abstract

Diffusion-based multimodal large language models (Diffusion MLLMs) have recently demonstrated impressive non-autoregressive generative capabilities across vision-and-language tasks. However, Diffusion MLLMs exhibit substantially slower inference than autoregressive models: Each denoising step employs full bidirectional self-attention over the entire sequence, resulting in cubic decoding complexity that becomes computationally impractical with thousands of visual tokens. To address this challenge, we propose D$^{3}$ToM, a Decider-guided dynamic token merging method that dynamically merges redundant visual tokens at different denoising steps to accelerate inference in Diffusion MLLMs. At each denoising step, D$^{3}$ToM uses decider tokens-the tokens generated in the previous denoising step-to build an importance map over all visual tokens. Then it maintains a proportion of the most salient tokens and merges the remainder through similarity-based aggregation. This plug-and-play module integrates into a single transformer layer, physically shortening the visual token sequence for all subsequent layers without altering model parameters. Moreover, D$^{3}$ToM employs a merge ratio that dynamically varies with each denoising step, aligns with the native decoding process of Diffusion MLLMs, achieving superior performance under equivalent computational budgets. Extensive experiments show that D$^{3}$ToM accelerates inference while preserving competitive performance. The code is released at https://github.com/bcmi/D3ToM-Diffusion-MLLM.

Figures (11)

• Figure 1: D3ToM Performance. Our D3ToM outperforms the current SOTA methods such as FastV, PDrop and VisionZip, achieving over 96% of the performance with only 10% of the visual tokens on LaViDa.
• Figure 2: Illustration of visual token redundancy. Probability density distribution of attention weights from output tokens to grouped visual tokens. Because an image contains over one thousand visual tokens while the decoder generates only 64 output tokens, every 16 consecutive visual tokens are averaged to form a single group, and attention from each output token is aggregated over these groups. The resulting distribution shows that most visual-token groups receive near-zero attention, indicating substantial redundancy.
• Figure 3: Visualization of attention weights. The figure displays four snapshots of attention weights at different denoising steps (8, 16, 24, and 32) assigned to visual tokens. Each snapshot shows attention distribution of the output tokens generated at the current denoising step on the input image of a small white Chihuahua dog on a wooden stump. The highlighted regions in each image represent the areas of the input image that receive the most attention from the output tokens generated at that specific decoding step. Different colors indicate the attention focus of output tokens generated at different steps.
• Figure 4: The architecture of the D3ToM framework. The main architecture (left) shows the merging operation occurring at layer $l^*$. The detailed process (right) illustrates the two key stages: (1) Decider-Guided Selection, where decider tokens guide the selection of visual tokens to be kept, and (2) Similarity-Guided Merging, where merging tokens are aggregated into their most similar kept tokens.
• Figure 5: FLOPs reduction (%) as a function of merge layer $l^{\ast}$ and merge ratio $\alpha$. Higher is better.