Better Prompt Compression Without Multi-Layer Perceptrons

TL;DR

This work tackles prompt compression for speeding up language model inference without altering the generator by challenging the assumption that encoders must mirror the decoder. It introduces the Attention-Only Compressor (AOC), an encoder built by removing MLP layers from Transformer blocks, achieving ~67% fewer parameters while maintaining or improving compression performance up to 480x compared to LoRA-based baselines. Empirical results show AOC outperforms the 500xCompressor across multiple prompt lengths and memory-token configurations, with latent-space interpolation revealing meaningful, smooth transitions between compressed prompts. The findings suggest that encoder architecture need not match the decoder’s, opening avenues for more efficient compression strategies and richer analysis of compressed latent representations.

Abstract

Prompt compression is a promising approach to speeding up language model inference without altering the generative model. Prior works compress prompts into smaller sequences of learned tokens using an encoder that is trained as a LowRank Adaptation (LoRA) of the inference language model. However, we show that the encoder does not need to keep the original language model's architecture to achieve useful compression. We introduce the Attention-Only Compressor (AOC), which learns a prompt compression encoder after removing the multilayer perceptron (MLP) layers in the Transformer blocks of a language model, resulting in an encoder with roughly 67% less parameters compared to the original model. Intriguingly we find that, across a range of compression ratios up to 480x, AOC can better regenerate prompts and outperform a baseline compression encoder that is a LoRA of the inference language model without removing MLP layers. These results demonstrate that the architecture of prompt compression encoders does not need to be identical to that of the original decoder language model, paving the way for further research into architectures and approaches for prompt compression.