HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing

TL;DR

HySparse introduces a simple yet effective hybrid attention architecture that interleaves full attention with sparse layers, using oracle token selection and cross-layer KV cache sharing to reduce memory and computation. By reusing the preceding full layer's token indices and KV caches, plus an independent SWA branch for local modeling, HySparse preserves long-context capabilities while achieving substantial KV cache savings. Across 7B dense and 80B MoE models, HySparse outperforms both full attention and Hybrid SWA baselines, with particularly large gains and memory reductions in the 80B MoE setting where only a small fraction of layers use full attention. The approach offers a practical path toward scalable, long-context LLMs by balancing global retrieval and local coherence with efficient KV cache management.

Abstract

This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simple, HySparse strategically derives each sparse layer's token selection and KV caches directly from the preceding full attention layer. This architecture resolves two fundamental limitations of prior sparse attention methods. First, conventional approaches typically rely on additional proxies to predict token importance, introducing extra complexity and potentially suboptimal performance. In contrast, HySparse uses the full attention layer as a precise oracle to identify important tokens. Second, existing sparse attention designs often reduce computation without saving KV cache. HySparse enables sparse attention layers to reuse the full attention KV cache, thereby reducing both computation and memory. We evaluate HySparse on both 7B dense and 80B MoE models. Across all settings, HySparse consistently outperforms both full attention and hybrid SWA baselines. Notably, in the 80B MoE model with 49 total layers, only 5 layers employ full attention, yet HySparse achieves substantial performance gains while reducing KV cache storage by nearly 10x.

Figures (2)

• Figure 1: HySparse Architecture Diagram. Each full attention layers is interleaved with multiple sparse attention layers. Sparse attention directly reuses the KV cache and important token indices from the preceding full attention layer.
• Figure 2: HySparse Accuracy vs. Training Iterations.