PoC: Performance-oriented Context Compression for Large Language Models via Performance Prediction

Abstract

While context compression can mitigate the growing inference costs of Large Language Models (LLMs) by shortening contexts, existing methods that specify a target compression ratio or length suffer from unpredictable performance degradation, hindering their reliable deployment. We introduce a paradigm shift to Performance-oriented Context Compression (PoC), where developers specify an acceptable performance floor instead of a compression ratio. PoC employs a lightweight performance predictor to automatically find the most aggressive compression ratio that satisfies this constraint before steering an off-the-shelf compressor. We design and compare two predictor variants: a simple context-agnostic predictor and a more sophisticated context-aware one that considers the input's inherent compressibility. On both question-answering and summarization benchmarks, the context-aware predictor consistently achieves lower performance prediction error than the context-agnostic predictor, while the resulting context-aware PoC attains a superior overall performance. Our work paves the way for a more reliable, efficient, and performance-aware deployment of context compression for LLMs.

Figures (5)

• Figure 1: Overview of Performance-oriented Context Compression (PoC). PoC is a novel context compression paradigm that moves away from budget-driven compression toward performance-driven compression. Unlike Budget-oriented Context Compression (BoC), PoC compresses context to the minimum length required to meet a given performance constraint. This approach mitigates the unpredictable performance degradation inherent in BoC, thus enhancing the reliability and applicability of context compression.
• Figure 2: Overview of the PoC pipeline: (1) A performance predictor first estimates the performance-compression curve for the input context. (2) Given a target performance retention, PoC then searches for the minimum compression ratio that meets the constraint. (3) Finally, an off-the-shelf compressor compresses the context using the selected ratio.
• Figure 3: Context-aware predictor. It takes the context and multiple compression ratios as input to simultaneously predict multiple performance retentions. This parallel prediction approach enables the prediction of multiple performance retentions in a single forward pass, significantly accelerating the prediction speed.
• Figure 4: The performance-compression curve interpolated by the context-agnostic predictor. To illustrate the sample-specific variability, we randomly plot the performance-compression curves of 3 samples. (a)--(f) present the performance-compression curves for the QA and summarization tasks, while (h) shows the unnormalized performance-compression curve (where the y-axis represents the F1 score) on TriviaQA. The unnormalized performance-compression curves for the other datasets are provided in Appendix \ref{['sec:scores']}.
• Figure 5: The unnormalized performance-compression curves interpolated by the context-agnostic predictor. In subfigures (a)--(e), the y-axis represents the F1 score, while in (f) and (g), the y-axis represents the geometric mean of ROUGE-1, ROUGE-2, and ROUGE-L.