Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference

TL;DR

Prompt length in LLM inference imposes high computational cost, motivating efficient prompt compression. CPC introduces a sentence-level, context-aware encoder trained with a contrastive objective on a newly built CQR dataset to rank context sentences by relevance to a given question, selecting the most informative sentences within a token budget. The approach yields state-of-the-art results on LongBench and ZeroSCROLLS, with up to a 10.93x reduction in inference latency and strong domain generalization, while preserving answer quality. The work provides substantial practical impact by enabling faster, more readable prompts and releasing the code and data to support reproducibility and further research.

Abstract

Large language models (LLMs) have triggered a new stream of research focusing on compressing the context length to reduce the computational cost while ensuring the retention of helpful information for LLMs to answer the given question. Token-based removal methods are one of the most prominent approaches in this direction, but risk losing the semantics of the context caused by intermediate token removal, especially under high compression ratios, while also facing challenges in computational efficiency. In this work, we propose context-aware prompt compression (CPC), a sentence-level prompt compression technique where its key innovation is a novel context-aware sentence encoder that provides a relevance score for each sentence for a given question. To train this encoder, we generate a new dataset consisting of questions, positives, and negative pairs where positives are sentences relevant to the question, while negatives are irrelevant context sentences. We train the encoder in a contrastive setup to learn context-aware sentence representations. Our method considerably outperforms prior works on prompt compression on benchmark datasets and is up to 10.93x faster at inference compared to the best token-level compression method. We also find better improvement for shorter length constraints in most benchmarks, showing the effectiveness of our proposed solution in the compression of relevant information in a shorter context. Finally, we release the code and the dataset for quick reproducibility and further development: https://github.com/Workday/cpc.

Figures (4)

• Figure 1: Overview of CPC. We first train a novel context-aware sentence encoder contrastively on a newly generated dataset of questions, positives, and negative pairs. Our model then uses this context-aware encoder to generate embeddings of the question and context sentences, and uses the similarity between them to select relevant sentences.
• Figure 2: Data collection pipeline. The data collection process starts by asking the LLM with Prompt 1 to generate a set of ($Q$,$A$) pairs. In the next step, we verify the pairs with Prompt 2. In the last step, we select a set of negatives for each ($Q$,$A$) pair.
• Figure 3: Comparison of latency between LongLLMLingua and our proposed method.
• Figure 4: An example illustrating the question and sentence relevance scores from our context-aware sentence encoder. Here, the more relevant sentences are coloured with higher intensity.