Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression

Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yukun Yan, Shuo Wang, Ge Yu

TL;DR

The paper tackles the inefficiency of long prompts in large language models by introducing Gist-COCO, a compression framework that learns gist representations of prompts with a dedicated encoder and uses them as prefixes to the input. Guided by an MDL-inspired objective, the compression module emulates the behavior of full prompts while keeping the base language model frozen, and a gist verbalization step enables cross-model applicability to decoder-based LMs. Empirical results show Gist-COCO outperforms prior prompt compression methods on both passage and instruction tasks, with analysis revealing that gist prompts can directly provide answers, support chain-of-thought, or repeat content depending on the task. The work advances efficient prompting and interpretability in prompt engineering, providing practical methods for reducing context length while preserving or enhancing model performance across diverse LLMs.

Abstract

Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. Gist-COCO employs an encoder-decoder based language model and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model. By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding language models. They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. All data and codes are available at https://github.com/OpenMatch/Gist-COCO .

Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression

TL;DR

The paper tackles the inefficiency of long prompts in large language models by introducing Gist-COCO, a compression framework that learns gist representations of prompts with a dedicated encoder and uses them as prefixes to the input. Guided by an MDL-inspired objective, the compression module emulates the behavior of full prompts while keeping the base language model frozen, and a gist verbalization step enables cross-model applicability to decoder-based LMs. Empirical results show Gist-COCO outperforms prior prompt compression methods on both passage and instruction tasks, with analysis revealing that gist prompts can directly provide answers, support chain-of-thought, or repeat content depending on the task. The work advances efficient prompting and interpretability in prompt engineering, providing practical methods for reducing context length while preserving or enhancing model performance across diverse LLMs.

Abstract

Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. Gist-COCO employs an encoder-decoder based language model and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model. By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding language models. They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. All data and codes are available at https://github.com/OpenMatch/Gist-COCO .
Paper Structure (17 sections, 9 equations, 5 figures, 7 tables)

This paper contains 17 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminary of Prompt Compression
  5. Prompt Compression via Gist Conditioned Decoding
  6. Compression Generalization for Different Prompts and Language Models
  7. Experimental Methodology
  8. Evaluation Results
  9. Overall Performance
  10. Ablation Studies
  11. Characteristics of Learned Gist Representations
  12. Case Studies
  13. Conclusion
  14. Appendix
  15. License
  16. ...and 2 more sections

Figures (5)

  • Figure 1: The Motivation of Our Gist Conditioned Decoding (Gist-COCO) Model. The user respectively utilizes promptsand compressed promptsto guide the generation of LLMs.
  • Figure 2: Training of Gist-COCO. Gist-COCO is trained to emulate the output distribution based on uncompressed inputs by producing gist representations.
  • Figure 3: Effectiveness of Gist Verbalization Results. We use different numbers of compression tokens.
  • Figure 4: Text Similarity between the Gist Verbalization Results with Inputs and Prompts.
  • Figure 5: Distribution of Categorizations of Gist Verbalization Results. We categorize Alpaca+ tasks into distinct groups and present the categorization outcomes of verbalization results across various tasks.