Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, Tongxuan Liu, Di yin, Xing Sun, Muhan Zhang

Abstract

Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a lightweight indexer, then computing attention only on the selected subset. While the downstream sparse attention itself scales favorably, the indexer must still scan the entire prefix for every query, introducing an per-layer bottleneck that grows prohibitively with context length. We propose HISA (Hierarchical Indexed Sparse Attention), a plug-and-play replacement for the indexer that rewrites the search path from a flat token scan into a two-stage hierarchical procedure: (1) a block-level coarse filtering stage that scores pooled block representations to discard irrelevant regions, followed by (2) a token-level refinement stage that applies the original indexer exclusively within the retained candidate blocks. HISA preserves the identical token-level top-sparse pattern consumed by the downstream Sparse MLA operator and requires no additional training. On kernel-level benchmarks, HISA achieves up to speedup at 64K context. On Needle-in-a-Haystack and LongBench, we directly replace the indexer in DeepSeek-V3.2 and GLM-5 with our HISA indexer, without any finetuning. HISA closely matches the original DSA in quality, while substantially outperforming block-sparse baselines.

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

Abstract

Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a lightweight indexer, then computing attention only on the selected subset. While the downstream sparse attention itself scales favorably, the indexer must still scan the entire prefix for every query, introducing an per-layer bottleneck that grows prohibitively with context length. We propose HISA (Hierarchical Indexed Sparse Attention), a plug-and-play replacement for the indexer that rewrites the search path from a flat token scan into a two-stage hierarchical procedure: (1) a block-level coarse filtering stage that scores pooled block representations to discard irrelevant regions, followed by (2) a token-level refinement stage that applies the original indexer exclusively within the retained candidate blocks. HISA preserves the identical token-level top-sparse pattern consumed by the downstream Sparse MLA operator and requires no additional training. On kernel-level benchmarks, HISA achieves up to speedup at 64K context. On Needle-in-a-Haystack and LongBench, we directly replace the indexer in DeepSeek-V3.2 and GLM-5 with our HISA indexer, without any finetuning. HISA closely matches the original DSA in quality, while substantially outperforming block-sparse baselines.

Paper Structure

This paper contains 28 sections, 11 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Block sparse attention.
  4. Token sparse attention.
  5. Hierarchical sparse attention.
  6. Preliminary
  7. Indexer in DSA.
  8. Sparse MLA in DSA.
  9. Method
  10. HISA: Hierarchical Indexed Sparse Attention
  11. Block partitioning and pooled keys.
  12. Stage 1: Block-level coarse filtering.
  13. Stage 2: Token-level refinement.
  14. Boundary behavior.
  15. Complexity Analysis
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Comparison of the DSA token-wise indexer (left) and our HISA hierarchical block-level coarse filter followed by token-level refinement (right). Both produce the same data structure---a per-query set of $k$ token indices---consumed by the downstream Sparse MLA operator.
  • Figure 2: Latency comparison of the indexer kernel between the original DSA (flat token scan) and HISA (hierarchical block-to-token indexing). In the left panel, the block size is fixed to $B=128$ and the maximum number of selected blocks is set to top-$m=64$. In the right panel, the block size is also fixed to $B=128$, while the number of selected blocks is adjusted for each sequence length to maintain a fixed compression ratio of $M\!:\!m = 4\!:\!1$.
  • Figure 3: Needle-in-a-Haystack retrieval accuracy heatmaps for DeepSeek-V3.2 under three indexing strategies. The $x$-axis denotes the context length (8K--128K), and the $y$-axis denotes the needle depth (0%--100%). Shades closer to green indicate higher retrieval accuracy.
  • Figure 4: Visualization of Attention Distribution.
  • Figure 5: LongBench scores under different indexer configurations. All three HISA variants use a candidate token pool of size $mB = 8192$ and a final token budget of $k{=}2048$, with different choices of block size $B$ and block-level top-$m$. The Block-Sparse baseline uses $B{=}128$ and $m{=}16$, corresponding to a candidate pool of 2048 tokens and no token-level refinement.