Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs

TL;DR

This paper tackles the problem of severe information loss when compressing long contexts for LLMs, particularly in multi-document QA. It introduces Query-Guided Compressor (QGC), a white-box module with four components—Query-Guided Context Encoder, Query-Guided Pooling, Query-Document Reviewing Layer, and Semantic Alignment Layer—plus a dynamic compression strategy, trained while keeping the LLM fixed. Empirical results on NaturalQuestions, TriviaQA, and HotpotQA show that QGC achieves substantially higher compression ratios and throughput with competitive or superior accuracy compared to state-of-the-art baselines, and ablations confirm the critical contribution of each component. The approach promises practical impact by reducing inference cost and latency in retrieval-augmented QA and in-context learning scenarios, while revealing avenues for extending to additional tasks and LLMs in future work.

Abstract

The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closed-book level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput.

Figures (6)

• Figure 1: The accuracy of LongLLMLingua jiang2023longllmlingua and AutoCompressor chevalier2023adapting with different compression ratios and number of documents on the NaturalQuestions test set, respectively. Closed-book denotes providing LLMs with the question only, and Oracle means using the question and corresponding ground-truth documents as the input of the LLM. "w/ answer" means adding the golden answer to the compressed context.
• Figure 2: The framework of our method.
• Figure 3: The structure of QGC. The first three layers use query $q$ to guide document $d$ encoding, pooling, and reviewing respectively. The last layer aligns document representations into the target LLM embedding space.
• Figure 4: The accuracy of QGC with varying compression ratios and number of documents, respectively.
• Figure 5: The accuracy, compression throughput, and generation throughput of QGC and LongLLMLingua.