EFPC: Towards Efficient and Flexible Prompt Compression

TL;DR

EFPC tackles the high cost of prompting large LLMs by unifying task-aware and task-agnostic prompt compression into a single, data-efficient framework. It combines instruction-aware data collection using GPT-4 to generate compressed prompts with a lightweight Transformer encoder that predicts per-token preservation, enabling selective truncation during training and inference. The method demonstrates clear gains: at a 4x compression rate, EFPC achieves a 4.8% relative F1 improvement on LongBench single-doc QA with only 1% more data, and 11.4% with 10% more data, outperforming LLMLingua-2; it also shows strong generalization across models and domains and improves latency/memory. These results indicate EFPC's practical potential for efficient, scalable prompt compression in real-world NLP systems.

Abstract

The emergence of large language models (LLMs) like GPT-4 has revolutionized natural language processing (NLP), enabling diverse, complex tasks. However, extensive token counts lead to high computational and financial burdens. To address this, we propose Efficient and Flexible Prompt Compression (EFPC), a novel method unifying task-aware and task-agnostic compression for a favorable accuracy-efficiency trade-off. EFPC uses GPT-4 to generate compressed prompts and integrates them with original prompts for training. During training and inference, we selectively prepend user instructions and compress prompts based on predicted probabilities. EFPC is highly data-efficient, achieving significant performance with minimal data. Compared to the state-of-the-art method LLMLingua-2, EFPC achieves a 4.8% relative improvement in F1-score with 1% additional data at a 4x compression rate, and an 11.4% gain with 10% additional data on the LongBench single-doc QA benchmark. EFPC's unified framework supports broad applicability and enhances performance across various models, tasks, and domains, offering a practical advancement in NLP.

Figures (6)

• Figure 1: Performance under varying compression rates and training data amounts. Our EFPC method achieves significant improvements with minimal training data and larger gains with higher compression rates.
• Figure 2: Our instruction used for data distillation: sending user instructions and the original text to GPT-4 for compression, and the compressed text is required to complete the user instructions. The highlighted part shows the difference between our method and LLMLingua-2 llmlingua2 in data collection.
• Figure 3: Compression ratio distribution on MeetingBank meetingbank. It shows that our collected dataset achieves a much higher compression ratio (5 times more) compared to LLMLingua-2.
• Figure 4: The proposed method. From left to right are the data collection, model training, and inference processes.
• Figure 5: Data Efficiency of our method. We keep the same amount of training data and set target token to 3,000 during compression.