Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective

TL;DR

This work tackles the Repeat Curse, the repetitive text generation observed when applying cache to diffusion-based multimodal LLMs. Guided by information-flow analysis, it shows that context tokens act as anchors that accumulate semantic information across layers, with the deep-layer entropy of these tokens $E_{sum}^{ctx}(i)$ converging under normal decoding; caching disrupts this flow, causing non-convergence and repetition. The authors propose CoTA, a training-free method consisting of Context Tokens Attention Enhancement (CTAE) and Context Tokens Entropy-Guided Voting (CTEV) to preserve information flow and penalize uncertain outputs, respectively. Across eight multimodal benchmarks and additional models, CoTA significantly mitigates repetition with modest overhead and generalizes to other cache methods and models, offering a practical path to more reliable cache-enabled dMLLMs.

Abstract

Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA

Figures (11)

• Figure 1: Motivation. When cache is applied to accelerate dMLLMs, the generated responses often exhibit excessive token repetition—a phenomenon we term the Repeat Curse.
• Figure 2: (a) presents the quantitative results of the “Repeat Curse”. L and S indicate evaluations on long-text responses (512 tokens) and short-text responses (64 tokens). The evaluation metrics ARR, MRL, ARL, and 95pRL are introduced in Section \ref{['metric']} and Appendix \ref{['appendix2']}. (b) visualizes the attention maps of LLaDA-V and LLaDA-V+Cache.
• Figure 3: Attention Maps Visualization of LLaDA-V. Based on LLaDA-V 8B with a generation length of 128 and 32 decoding steps, we visualize the attention matrices corresponding to the masked (response) tokens, with the x-axis denoting key tokens and the y-axis denoting query tokens. Brighter colors indicate higher attention values. Context tokens act as anchors to aggregate information across layers and absorb attention.
• Figure 4: (a) and (b) correspond to context tokens from normal decoding and from decoding with repeated tokens, respectively. We define the set of context tokens as the target tokens together with their two nearest neighboring tokens in relative position. Context tokens with repetition tend to exhibit high entropy in deeper layers.
• Figure 5: Illustration of context tokens attention enhancement. Example values are annotated on the decay matrix for clarity, while actual values are computed using Equation \ref{['2']} and \ref{['1']}.