Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Sparse Attention: Access Patterns and Architecture

Noam Levy

Abstract

Dynamic sparse attention (DSA) reduces the per-token attention bandwidth by restricting computation to a top-k subset of cached key-value (KV) entries, but its token-dependent selection pattern introduces a system-level challenge: the KV working set is fragmented, volatile, and difficult to prefetch, which can translate into poor cache locality and stalled decode throughput. We study these effects by implementing a lightweight indexer for DSA-style selection on multiple open-source backbones and logging per-layer KV indices during autoregressive decoding. Our analysis shows a gap in serving DSA backbones - a potential for a high volume of blocking LL (last level) cache miss events, causing inefficiency; we propose a novel LL cache reservation system to save KV tokens in the LL cache between decode steps, combined with a token-granularity LRU eviction policy, and show on the data we collected how this architecture can benefit serving with DSA implemented on different backbones. Finally, we propose directions for future architectural and algorithmic exploration to improve serving of DSA on modern inference platforms.

Dynamic Sparse Attention: Access Patterns and Architecture

Abstract

Dynamic sparse attention (DSA) reduces the per-token attention bandwidth by restricting computation to a top-k subset of cached key-value (KV) entries, but its token-dependent selection pattern introduces a system-level challenge: the KV working set is fragmented, volatile, and difficult to prefetch, which can translate into poor cache locality and stalled decode throughput. We study these effects by implementing a lightweight indexer for DSA-style selection on multiple open-source backbones and logging per-layer KV indices during autoregressive decoding. Our analysis shows a gap in serving DSA backbones - a potential for a high volume of blocking LL (last level) cache miss events, causing inefficiency; we propose a novel LL cache reservation system to save KV tokens in the LL cache between decode steps, combined with a token-granularity LRU eviction policy, and show on the data we collected how this architecture can benefit serving with DSA implemented on different backbones. Finally, we propose directions for future architectural and algorithmic exploration to improve serving of DSA on modern inference platforms.
Paper Structure (19 sections, 6 equations, 9 figures, 4 tables)

This paper contains 19 sections, 6 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. The Lightning Indexer
  4. Architectural Analysis
  5. Profiling
  6. Analysis Results
  7. Working Set Size
  8. KV Entry Persistence
  9. Lookback Steps
  10. New Lookup Count
  11. Inter-layer Index Overlap
  12. Inter-layer Variations
  13. KV-granular Last Level (LL) Cache
  14. Additional Architectural Directions for Future Work
  15. DSA and paged attention
  16. ...and 4 more sections

Figures (9)

  • Figure 1: Dynamic Sparse Attention overview; an indexer, using a small number of heads and small projection dimensionality, scores each context token; the top-k score are selected, and a the KV values corresponding are gathered from the KV cache, which is typically stored in HBM memory; the scaled dot product attention is applied only to this subset
  • Figure 2: Blockwise sparse attention vs. DSA sparse attention; each query may select a completely different top-$k$ KV subset to attend to, eliminating the reuse potential of previous approaches and introducing an implementation challenge.
  • Figure 3: Distribution of the number of KV entries in a working set of a 50 consecutive tokens, as a fraction of top-k
  • Figure 4: Distribution of persistence over time steps
  • Figure 5: Distribution of lookback distance, over fraction of top-k
  • ...and 4 more figures