QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization

TL;DR

The paper tackles the problem of ultra-long-context processing in LLMs, where quadratic computation and the 'lost in the middle' phenomenon hinder performance. It presents QwenLong-CPRS, a dynamic context optimization framework with token-level critics, bidirectional location reasoning, and window-parallel inference to compress long inputs into query-relevant content. Across five long-context benchmarks, it achieves up to $21.59×$ context compression and $19.15$-point average gains, while remaining architecture-agnostic and enabling smaller LLMs to rival larger ones in ultra-long contexts, including new SOTA results on several tasks. The method yields linear latency scaling and substantial speedups (e.g., $3.47×$ acceleration at 128K) and is compatible with both customized prompts and standard prompting, offering practical benefits for deploying long-context LLMs in real-world settings.

Abstract

This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance. Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference. Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59$\times$ context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance.

Figures (10)

• Figure 1: Illustration of the performance of QwenLong-CPRS. Figure \ref{['fig:intro_compress_and_performance']} compares the input token consumption and model performance of various LLMs on Ruler-128K before (marked with $\Diamond$) and after (marked with $\Delta$) cascading QwenLong-CPRS. Figure \ref{['fig:intro_method_compare']} highlights the performance improvements of QwenLong-CPRS over other context management methods, such as RAG gteembedding and sparse attention jiang2024minference.
• Figure 2: The concept of dynamic context optimization, which aims to enhance context processing efficiency by maximizing information density. Given a long-context input, this paradigm dynamically compresses it into query-specific content at varying granularities, facilitating concise and accurate information extraction for different user queries. For instance, keywords for search queries, sentences for question answering, and paragraphs for summarization.
• Figure 3: The model architecture and workflow of QwenLong-CPRS.
• Figure 4: Performance of QwenLong-CPRS in NIAH test with input length is upto 1M.
• Figure 5: Comparative performance analysis of LLMs with and without QwenLong-CPRS integration. Numerical values in parentheses indicate each model's maximum input capacity.