Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern

TL;DR

This work tackles the quadratic cost of full attention in long-context LLMs by uncovering localized triangular attention patterns and introducing Ltri-LLM, a training-free framework that identifies semantic spans, stores them in an offline span index, and retrieves relevant spans in memory during streaming inference. Key components include triangle-based span discovery with NMS, static and dynamic span index vectors for efficient retrieval, and a collaborative retrieval mechanism with persistent recall and retrieval-head voting to boost evidence recall. Extensive experiments on NIAH, ∞-Bench, and RULER show Ltri-LLM achieving near FA performance while maintaining streaming efficiency, with strong retrieval capabilities and robust ablation results supporting the design choices. The approach offers a practical path toward scalable long-context inference in open-source LLMs by combining local attention structure, span-level indexing, and evidence-driven retrieval without additional training.

Abstract

The quadratic computational complexity of the attention mechanism in current Large Language Models (LLMs) renders inference with long contexts prohibitively expensive. To address this challenge, various approaches aim to retain critical portions of the context to optimally approximate Full Attention (FA) through Key-Value (KV) compression or Sparse Attention (SA), enabling the processing of virtually unlimited text lengths in a streaming manner. However, these methods struggle to achieve performance levels comparable to FA, particularly in retrieval tasks. In this paper, our analysis of attention head patterns reveals that LLMs' attention distributions show strong local correlations, naturally reflecting a chunking mechanism for input context. We propose Ltri-LLM framework, which divides KVs into spans, stores them in an offline index, and retrieves the relevant KVs into memory for various queries. Experimental results on popular long text benchmarks show that Ltri-LLM can achieve performance close to FA while maintaining efficient, streaming-based inference.

Figures (24)

• Figure 1: Average Needle-In-A-Haystack performance of InfLLM on a set of NIAH tests consisting of eight questions. More details can be found in Appendix \ref{['appendix::fan']}.
• Figure 2: The average needle recall across layers and decoding steps of successful and failed NIAH cases.
• Figure 3: Triangular attention regions indicate semantic segments.
• Figure 4: Cumulative scores calculation for each span. First, minus a threshold $\theta$. Second, sum the upper and right elements.
• Figure 5: Three metrics gauge the confidence of the voters, i.e., the ratio between the span and its row neighbors, its column neighbors and its row+column neighbors.