Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling

Qihang Fan, Huaibo Huang, Zhiying Wu, Juqiu Wang, Bingning Wang, Ran He

TL;DR

This paper proposes FlashPrefill, a framework enabling ultra-fast prefilling via instantaneous pattern discovery and thresholding that introduces a dynamic thresholding mechanism that bypasses the prohibitive overhead of sorting or accumulating attention scores while effectively eliminating the long-tail distribution to enhance sparsity.

Abstract

Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. While various sparse attention mechanisms have been explored, they typically suffer from either significant search latency or insufficient sparsity. In this paper, we propose FlashPrefill, a framework enabling ultra-fast prefilling via instantaneous pattern discovery and thresholding. FlashPrefill leverages a fast block-searching technique to simultaneously locate dynamic vertical, slash, and block-sparse attention patterns. Crucially, it introduces a dynamic thresholding mechanism that bypasses the prohibitive overhead of sorting or accumulating attention scores while effectively eliminating the long-tail distribution to enhance sparsity. Extensive evaluations demonstrate that FlashPrefill achieves a substantial leap in efficiency, delivering an unprecedented 27.78x speedup on 256K sequences. Notably, unlike existing methods that incur efficiency degradation on shorter contexts, FlashPrefill maintains a 1.71x speedup even at a 4K context length, demonstrating its robustness and practical utility across varying sequence scales.

FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling

TL;DR

This paper proposes FlashPrefill, a framework enabling ultra-fast prefilling via instantaneous pattern discovery and thresholding that introduces a dynamic thresholding mechanism that bypasses the prohibitive overhead of sorting or accumulating attention scores while effectively eliminating the long-tail distribution to enhance sparsity.

Abstract

Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. While various sparse attention mechanisms have been explored, they typically suffer from either significant search latency or insufficient sparsity. In this paper, we propose FlashPrefill, a framework enabling ultra-fast prefilling via instantaneous pattern discovery and thresholding. FlashPrefill leverages a fast block-searching technique to simultaneously locate dynamic vertical, slash, and block-sparse attention patterns. Crucially, it introduces a dynamic thresholding mechanism that bypasses the prohibitive overhead of sorting or accumulating attention scores while effectively eliminating the long-tail distribution to enhance sparsity. Extensive evaluations demonstrate that FlashPrefill achieves a substantial leap in efficiency, delivering an unprecedented 27.78x speedup on 256K sequences. Notably, unlike existing methods that incur efficiency degradation on shorter contexts, FlashPrefill maintains a 1.71x speedup even at a 4K context length, demonstrating its robustness and practical utility across varying sequence scales.
Paper Structure (26 sections, 4 equations, 7 figures, 9 tables, 3 algorithms)

This paper contains 26 sections, 4 equations, 7 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large Language Models.
  4. Sparse Attention.
  5. Method
  6. Instantaneous Pattern Discovery
  7. Block Approximation for Kernel Optimization
  8. Fused Block-level Attention Approximation.
  9. Consistency-preserved Global Normalization.
  10. Comparison with Previous Methods.
  11. Max-based Dynamic Thresholding
  12. Optimized Block Sparse Attention Kernel
  13. Experiments
  14. Accuracy and Efficiency Results
  15. Baselines.
  16. ...and 11 more sections

Figures (7)

  • Figure 1: "Needle In A Haystack" evaluation of Qwen3-30B-A3B-Instruct-2507 using FlashPrefill across context lengths ranging from 2K to 256K.
  • Figure 2: End-to-end Time-to-First-Token (TTFT) speedup relative to full attention on Qwen3-30B-A3B-Instruct-2507 within the vLLM framework.
  • Figure 3: Comparative speedup of various operators relative to Flash Attention flashattention2 on Qwen3-30B-A3B-Instruct-2507. FlashPrefill exhibits a dominant advantage, particularly in long-context scenarios.
  • Figure 4: Execution time of different parts across different approaches. All results are measured on Qwen3-30B-A3B-Instruct-2507.
  • Figure 5: Illustration of pattern discovery. Red dashed lines represent the uniformly distributed queries.
  • ...and 2 more figures