Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

Warren Johnson

Abstract

Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substantially lower expansion on HumanEval (5x, Psi approx 0.72), while GPT-4o-mini is comparatively stable across benchmarks. This reconciles the apparent discrepancy between previously reported extreme explosion and lower replication effects by identifying prompt structure, not provider identity alone, as the primary moderator. We introduce the Compression Robustness Index (CRI) for cross-benchmark evaluation and show that single-benchmark assessments can produce misleading conclusions about compression safety and efficiency. To contextualize energy claims, we incorporate companion direct NVML measurements from rented RunPod GPUs and show that token savings can overstate joule savings. These findings motivate benchmark-diverse testing and structure-aware compression policies for reliable, energy-conscious LLM deployment.

Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

Abstract

Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substantially lower expansion on HumanEval (5x, Psi approx 0.72), while GPT-4o-mini is comparatively stable across benchmarks. This reconciles the apparent discrepancy between previously reported extreme explosion and lower replication effects by identifying prompt structure, not provider identity alone, as the primary moderator. We introduce the Compression Robustness Index (CRI) for cross-benchmark evaluation and show that single-benchmark assessments can produce misleading conclusions about compression safety and efficiency. To contextualize energy claims, we incorporate companion direct NVML measurements from rented RunPod GPUs and show that token savings can overstate joule savings. These findings motivate benchmark-diverse testing and structure-aware compression policies for reliable, energy-conscious LLM deployment.
Paper Structure (39 sections, 1 theorem, 9 equations, 1 figure, 6 tables)

This paper contains 39 sections, 1 theorem, 9 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Prompt Compression Techniques
  4. LLM Energy Consumption and Green AI
  5. Code Generation Benchmarks and Evaluation
  6. Output Behavior Under Input Perturbation
  7. Theoretical Framework
  8. Experimental Methodology
  9. Dataset Composition
  10. Compression Protocol
  11. Model Selection
  12. Statistical Framework
  13. Energy Model
  14. Companion Direct Joule Calibration
  15. Results
  16. ...and 24 more sections

Key Result

Proposition 1

Let $\mathbf{x}^{\text{MBPP}}$ and $\mathbf{x}^{\text{HumanEval}}$ denote typical prompts from the MBPP and HumanEval benchmarks respectively. Under first-$N$-words compression at ratio $r=0.3$:

Figures (1)

  • Figure 1: Output token count vs. instruction survival probability $\Psi$ for DeepSeek-Chat at $r=0.3$, grouped by benchmark. MBPP prompts (low $\Psi$) cluster near the 1024-token ceiling, while HumanEval prompts (high $\Psi$) produce focused outputs. The threshold $\tau \approx 0.35$ separates ceiling-hitting behavior from linear compensation. The continuous piecewise model captures the transition between regimes.

Theorems & Definitions (4)

  • Definition 1: Instruction Segment
  • Definition 2: Instruction Survival Probability
  • Proposition 1: Benchmark-Dependent Survival
  • Definition 3: Compression Robustness Index