An Empirical Study on Prompt Compression for Large Language Models

TL;DR

This work tackles the cost and latency associated with long prompts for large language models by conducting an empirical study of six prompt compression methods across 13 datasets and three (M)LLMs, introducing the compression ratio $\rho = 1 - \frac{L_c}{L_o}$ to quantify input shortening. It evaluates six compressors (KiS, SCRL, Selective Context, LLMLingua, LongLLMLingua, LLMLingua-2) on tasks including summarization, reconstruction, and QA, with multimodal VQA analysis, and uses MiHR/MaHR to assess hallucinations. The authors find long-context prompts are more sensitive to compression, moderate compression can improve long-context QA, and LLMLingua variants generally deliver the best performance at high compression, while all methods increase hallucinations due to information loss. They also introduce PCToolkit for reproducible evaluation and provide insights on word omission and multimodal applicability, underscoring practical guidance for prompt design in real-world deployments.

Abstract

Prompt engineering enables Large Language Models (LLMs) to perform a variety of tasks. However, lengthy prompts significantly increase computational complexity and economic costs. To address this issue, we study six prompt compression methods for LLMs, aiming to reduce prompt length while maintaining LLM response quality. In this paper, we present a comprehensive analysis covering aspects such as generation performance, model hallucinations, efficacy in multimodal tasks, word omission analysis, and more. We evaluate these methods across 13 datasets, including news, scientific articles, commonsense QA, math QA, long-context QA, and VQA datasets. Our experiments reveal that prompt compression has a greater impact on LLM performance in long contexts compared to short ones. In the Longbench evaluation, moderate compression even enhances LLM performance. Our code and data is available at https://github.com/3DAgentWorld/Toolkit-for-Prompt-Compression.

Figures (11)

• Figure 1: Illustration of prompt compression. The original context is distilled into a more concise form while preserving pertinent information for LLMs to process. Some methods compress the context based on the query, while others do not. Words that are underlined in the original text denote the segments that are trimmed by the compressor.
• Figure 2: Categories of prompt compression methods. These methods can be grouped into three main categories: (a) RL-based methods, which use heuristic rewards to optimize the compressor, (b) LLM scoring-based methods, which use another language model to score each token in a single autoregressive step and decide to keep or discard each token based on its score, and (c) LLM annotation-based methods, which use LLMs to annotate data for training a small model specifically designed for prompt compression.
• Figure 3: Computational overhead for different prompt compression methods. "Time per token" refers to the time taken divided by the number of tokens removed. All metrics are evaluated on a single A6000 GPU with 48 GB memory.
• Figure 3: Performance on compression tasks under different compression ratios. We measured the performance of four compression methods by changing the compression ratio while keeping all other settings in accordance with Table \ref{['tab:main']}. For each dataset, we randomly sampled 100 instances for evaluation and averaged their metrics. As mentioned in Section \ref{['implementations']}, KiS and SCRL cannot adjust the compression ratio and are thus not considered.
• Figure 4: Performance on different QA categories. We categorized the QA tasks into four categories: logic (Boolean Expression, Web of Lies), common sense (Causal Judgement), math (GSM8K), and long context (LongBench), and calculated the average performance of six prompt compression methods on these four categories. Considering the different metrics, we scaled the results based on the mean performance for each task.