Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

TL;DR

Adamas tackles the latency of self-attention in ultra-long contexts by introducing a token-level sparse-attention mechanism that leverages a Walsh-Hadamard transform, 2-bit bucketization, and a lightweight Manhattan-distance estimator to dynamically select relevant KV pairs. This approach preserves accuracy close to full attention at constrained budgets (e.g., $64$–$128$ tokens) and achieves up to $8\times$ higher sparsity with substantial speedups, including up to $4.4\times$ self-attention and $1.5\times$ end-to-end on $32$K sequences, while even attaining perplexity comparable or better than full attention. The method is training-free and relies on efficient GPU kernels for Hadamard, compression, and distance estimation, enabling practical deployment for long-context inference. Overall, Adamas offers a principled, efficient pathway to scaling LLMs to very long contexts without retraining or significant accuracy loss.

Abstract

Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods alleviate these costs but rely on heuristic patterns that struggle to recall critical key-value (KV) pairs for each query, resulting in accuracy degradation. We introduce Adamas, a lightweight yet highly accurate sparse attention mechanism designed for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and leverages Manhattan-distance estimation for efficient top-k selections. Experiments show that Adamas matches the accuracy of full attention with only a 64-token budget, achieves near-lossless performance at 128, and supports up to 8x higher sparsity than prior state-of-the-art (SOTA) methods while delivering up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences. Remarkably, Adamas attains comparable or even lower perplexity than full attention, underscoring its effectiveness in maintaining accuracy under aggressive sparsity.

Figures (6)

• Figure 1: Illustration of Adamas compared with existing methods. While StreamingLLM employs a fixed sparse pattern and Quest inherently selects pages, Adamas dynamically selects KV pairs at the token level, thereby achieving better preservation of model accuracy and inference efficiency.
• Figure 2: Overview of Adamas. Queries $Q$ and keys $K$ are processed through Hadamard transform, bucketization, and 2-bit compression. The transformed keys $\mathbf{H}_K$ are then compared against the transformed query $\mathbf{H}_Q$ in Manhattan-distance estimator, based on which the top-$k$ KV pairs are selected. Finally, Adamas performs sparse attention using $Q$ and the selected KV pairs.
• Figure 3: Evaluation results on LongBench. Adamas exhibits the smallest performance drop compared to full attention while maintaining high sparsity.
• Figure 4: Perplexity results of StreamingLLM, Quest, Adamas, and full attention evaluated on PG19 with LongChat-7b-v1.5-32k under varying token budgets. Adamas consistently matches full attention and even shows lower perplexity at larger token budget.
• Figure 5: Kernel-level breakdown of Adamas self-attention compared to full attention (FlashInfer).