Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage

TL;DR

This work identifies a paradox in post-training sparse-attention for long-decode inference: while sparse attention speeds per-step computation, it often leads to longer, information-poor generations, a phenomenon termed Lil. It establishes an information-theoretic lens using LZ77 compression to quantify information content and demonstrates that information loss in sparse decoding causes redundant reconstruction. To address this, the authors propose Guardian, an early-stopping algorithm that halts decoding when information gain stagnates, achieving up to 90% token savings with minimal accuracy loss on reasoning benchmarks. The results suggest Guardian not only improves efficiency for sparse decoding but also generalizes to prolong CoT generation, offering practical benefits for scalable, long-context LLM applications.

Abstract

Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency. Prior work typically decomposes inference into prefill and decode stages, with the decode stage dominating total latency. To reduce time and memory complexity in the decode stage, a line of work introduces sparse-attention algorithms. In this paper, we show, both empirically and theoretically, that sparse attention can paradoxically increase end-to-end complexity: information loss often induces significantly longer sequences, a phenomenon we term ``Less is Less'' (Lil). To mitigate the Lil problem, we propose an early-stopping algorithm that detects the threshold where information loss exceeds information gain during sparse decoding. Our early-stopping algorithm reduces token consumption by up to 90% with a marginal accuracy degradation of less than 2% across reasoning-intensive benchmarks.

Figures (5)

• Figure 1: Accuracy/output length vs. cache budget for five algorithms (legends) across three datasets (rows) and three models (columns). DSR, DSL, and Qwe denote DeepScaleR-1.5B-Preview, DeepSeek-R1-Distill-Llama-8B, and Qwen1.5-MoE-A2.7B-Chat, respectively. The x-axis shows varying cache budgets. In (a), the y-axis shows the proportion of correctly solved problems over 200 test cases. In (b), the y-axis shows the average output length over the same 200 test cases. For sparse-attention algorithms, the maximum generation length is capped at twice that of the full-attention baseline to prevent non-terminating generation.
• Figure 2: Compression ratio vs. cache budget. Notations follow Figure \ref{['fig-study-accuracy-length']}. The y-axis shows the average compression ratio (compressed-sequence length / original-sequence length) over 200 test cases.
• Figure 3: Average compressed length vs. original length for (a) sink (1024 cache budget) and (b) Quest (1024 cache budget) across three models and three datasets. Each line indicates an individual test case, with green and red curves denoting correct and incorrect outcomes, respectively. Other notations follow Figure \ref{['fig-study-accuracy-length']}. As the model generates more tokens (i.e., as the original length increases), the compressed length initially grows rapidly and then plateaus. Green curves denote correct test cases, whereas red curves denote incorrect test cases.
• Figure 4: Illustration of the LZ77 algorithm.
• Figure 5: Examples of information loss and gain.