In-Context Former: Lightning-fast Compressing Context for Large Language Model

TL;DR

The In-Context Former (IC-Former) model effectively reduces compression costs and makes real-time compression scenarios feasible and significantly reduces inference time, which achieves linear growth in time complexity within the compression range.

Abstract

With the rising popularity of Transformer-based large language models (LLMs), reducing their high inference costs has become a significant research focus. One effective approach is to compress the long input contexts. Existing methods typically leverage the self-attention mechanism of the LLM itself for context compression. While these methods have achieved notable results, the compression process still involves quadratic time complexity, which limits their applicability. To mitigate this limitation, we propose the In-Context Former (IC-Former). Unlike previous methods, IC-Former does not depend on the target LLMs. Instead, it leverages the cross-attention mechanism and a small number of learnable digest tokens to directly condense information from the contextual word embeddings. This approach significantly reduces inference time, which achieves linear growth in time complexity within the compression range. Experimental results indicate that our method requires only 1/32 of the floating-point operations of the baseline during compression and improves processing speed by 68 to 112 times while achieving over 90% of the baseline performance on evaluation metrics. Overall, our model effectively reduces compression costs and makes real-time compression scenarios feasible.

Figures (9)

• Figure 1: Compressing long contexts into short soft prompts (vectors in embedding space) to improve inference efficiency.
• Figure 2: Left: Model architecture of In-Context Former. In-Context Former utilizes a set of learnable digest embeddings to condense the information of context and generates digest vectors. And we apply causal attention masks for digest tokens. Right: Overview of In-Context Former's framework.
• Figure 3: Left: Pretraining stage. IC-Former learns to generate digest vectors such that, when these vectors and a special token AE are jointly fed into an LLM, the LLM reproduces the original context. Right: Instruction fine-tuning stage. Training IC-Former to generate digest vectors capable of correctly responding to prompts.
• Figure 4: The excessively long contexts are broken into chunks, which are then compressed and concatenated.
• Figure 5: BLEU-4 for different digest token lengths $k$.