Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor

TL;DR

This work tackles the rising costs of black-box LLM usage by proposing a prompt-compression pipeline where smaller LLMs compress inputs for larger ones. It introduces CompressionBench, a comprehensive benchmark across 25 models, and Cmprsr, a compression framework built on supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to achieve abstractive, token-level, question-agnostic compression that adheres to a target compression rate ($CR$). Through extensive QA and summarization evaluations on MeetingBank, GSM8K, and LongBench, Cmprsr demonstrates superior downstream performance while maintaining strict CR adherence ($\Delta_{CR} \leq 0$ in practice) and generalizes across input lengths and domains. The paper also shows how TextGrad can further improve vanilla compressors and discusses practical cost savings and latency considerations, highlighting a viable path for deploying smaller LLMs as effective prompt compressors at scale.

Abstract

Motivated by the high costs of using black-box Large Language Models (LLMs), we introduce a novel prompt compression paradigm, under which we use smaller LLMs to compress inputs for the larger ones. We present the first comprehensive LLM-as-a-compressor benchmark spanning 25 open- and closed-source models, which reveals significant disparity in models' compression ability in terms of (i) preserving semantically important information (ii) following the user-provided compression rate (CR). We further improve the performance of gpt-4.1-mini, the best overall vanilla compressor, with Textgrad-based compression meta-prompt optimization. We also identify the most promising open-source vanilla LLM - Qwen3-4B - and post-train it with a combination of supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), pursuing the dual objective of CR adherence and maximizing the downstream task performance. We call the resulting model Cmprsr and demonstrate its superiority over both extractive and vanilla abstractive compression across the entire range of compression rates on lengthy inputs from MeetingBank and LongBench as well as short prompts from GSM8k. The latter highlights Cmprsr's generalizability across varying input lengths and domains. Moreover, Cmprsr closely follows the requested compression rate, offering fine control over the cost-quality trade-off.

Figures (10)

• Figure 1: Extractive compression selects a subset of the input sequence tokens, while abstractive compression allows for clever paraphrases. While one can use vanilla LLMs as abstractive compressors, their performance can be further improved with RL-based post-training, yielding Cmprsr. Note that extractive compression may introduce ambiguities, e.g. "February April", "mission forward 50 years" bbc2025, while vanilla abstractive compression does not adhere to the desired compression rate.
• Figure 2: MeetingBank evaluation pipeline. We assess transcripts' compressions on $2$ downstream tasks. (i) Summarization, i.e we compute BertScore between compressed and original. (ii) QA, where we build a dataset of questions and answers from MeetingBank transcripts, and measure the Target model’s accuracy using the compressed context.
• Figure 3: TextGrad system prompt optimization for Qwen3-4B.
• Figure 4: TextGrad prompt optimization for gpt-4.1-mini.
• Figure 5: Distribution of $\Delta_{CR}$ across different models, illustrating adherence to the target CR.