LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

TL;DR

This work tackles task-agnostic prompt compression to improve LLM prompt generalizability and efficiency by learning from data distilled with LLM guidance.It introduces a data distillation pipeline that creates an extractive compression dataset from MeetingBank and formalizes compression as a token classification problem using a bidirectional Transformer encoder.The approach yields faithful, compact prompts that generalize across diverse domains and LLMs, delivering substantial latency reductions and competitive performance on both in-domain and out-of-domain tasks.A public dataset and code accompany the method, with demonstrated improvements over strong baselines and robust cross-domain transfer.

Abstract

This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x. Our code is available at https://aka.ms/LLMLingua-2.

Figures (16)

• Figure 1: Overview of LLMLingua-2.
• Figure 2: Our instruction used for data distillation.
• Figure 3: Distribution of compression ratio after chunk-wise compression on MeetingBank.
• Figure 4: Illustration of compression ratio w.r.t. original context length on MeetingBank. We use GPT-4-32k with the output token limit setting to 4096.
• Figure 5: Challenges in data annotation. (i) Ambiguity: a word in the compressed texts may appear multiple times in the original content. (ii) Variation: GPT-4 may modify the original words in tense, plural form, etc. during compression. (iii) Reordering: The order of words may be changed after compression.