LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs

TL;DR

LCIRC extends the context window of LLMs by recurrently compressing long-form input into compact representations and reinjecting them via gated cross-attention, enabling efficient processing of sequences far beyond native limits. The framework is augmented with Query Dependent LCIRC (QD-LCIRC), which conditionally preserves query-relevant information through an additional gating mechanism, improving performance on long-context tasks. Empirical results show substantial perplexity improvements and strong gains on ultra-long benchmarks (e.g., InfiniteBench, LongBench) compared to baselines like ExtendedFA and AutoCompressor, with QD-LCIRC delivering the best average performance. The contributions offer a scalable approach to long-context reasoning with practical implications for document understanding, long-form QA, and other tasks requiring extensive context and precise query relevance.

Abstract

While large language models (LLMs) excel in generating coherent and contextually rich outputs, their capacity to efficiently handle long-form contexts is limited by fixed-length position embeddings. Additionally, the computational cost of processing long sequences increases quadratically, making it challenging to extend context length. To address these challenges, we propose Long-form Context Injection with Recurrent Compression (LCIRC), a method that enables the efficient processing long-form sequences beyond the model's length limit through recurrent compression without retraining the entire model. We further introduce query dependent context modeling, which selectively compresses query-relevant information, ensuring that the model retains the most pertinent content. Our empirical results demonstrate that Query Dependent LCIRC (QD-LCIRC) significantly improves LLM's ability to manage extended contexts, making it well-suited for tasks that require both comprehensive context understanding and query relevance.

Figures (3)

• Figure 1: The overall process of the proposed Long-form Context Injection with Recurrent Compression (LCIRC). LCIRC comprises two components: Recurrent Context Compression (left) and Compressed Context Injection (right). In the $i$-th step of Recurrent Context Compression, the previously compressed features $\mathbf{h}^{(i-1)}$ and the segment embeddings $\mathbf{s}_{i}$ are fed into the Perceiver module as query and input features, respectively. The compressed features $\mathbf{h}^{(i)}$ are then generated and reinjected as query features for the subsequent recurrence step. The initial query features $\mathbf{h}^{(0)}$ are learnable parameters. In Compressed Context Injection, the concatenated compressed features $\mathbf{h}$ serve as input to the Gated Cross Attention layer. Layers indicated with a fire symbol represent trained layers, while layers marked with a snow symbol denote frozen layers.
• Figure 2: Comparison of the recurrent context compression module with and without query dependent modeling. In addition to the regular context compression module (left), we add additional cross attention module (blue box) to inject query information into the compressed feature $\mathbf{h}^{(i-1)}$ (right).
• Figure 3: Comparisons of the proposed Selective State BPTT with vanilla and truncated BPTT. Green boxes represent timesteps where gradients are computed in BPTT whereas the light green ones indicate the timesteps without gradient computation. Finally, dotted red lines illustrate the gradient flows. (a) Vanilla BPTT computes the full gradients through the entire timesteps in recurrence but is computationally infeasible with a large $N$. The gradients for $\mathbf{h}^{(i)}$ receives upstream gradients both through the recurrent connection and through the direct connection from $\mathbf{h}$. (b) Truncated BPTT backprobagates gradients to the last $T$ timesteps only significantly reducing computational costs. However, it does not transfer gradient flows to timesteps further than $T$ (marked with light green color) and fails to learn long-term QD modeling. (c) Our proposed Selective State BPTT selects several random timesteps and transfer gradient flows directly through the direct connection from $\mathbf{h}$, which enables efficient learning of long-term QD modeling capabilities.