DeCAL Tokenwise Compression
Sameer Panwar
TL;DR
DeCAL presents a tokenwise compression method that prepends a learned latent sequence to the encoder input and trains an encoder–decoder LM with a denoising objective to maximize compressed representation quality. With compression ratios from $2\times$ to $8\times$, DeCAL can match uncompressed baselines on several tasks at $2\times$, and incur only modest declines at $8\times$ across summarization, question answering, and multi-vector retrieval, while enabling significant storage and compute savings. The approach relies on a multi-layer latent cross-attention mechanism, initial pooling to seed the latent tokens, and denoising pretraining, outperforming proxy context-compression baselines and AttnPool variants. The work lays the groundwork for broader applicability, including potential future extensions to longer contexts, retrieval-augmented generation, and multimodal sources.
Abstract
This paper introduces DeCAL, a new method for tokenwise compression. DeCAL uses an encoder-decoder language model pretrained with denoising to learn to produce high-quality, general-purpose compressed representations from the encoder. DeCAL applies small modifications to the encoder, with the emphasis on maximizing compression quality, even at the expense of compute. We show that DeCAL at 2x compression can match uncompressed on several downstream tasks, with usually only a minor dropoff in metrics up to 8x compression, among question-answering, summarization, and multi-vector retrieval tasks. DeCAL offers significant savings where pre-computed dense representations can be utilized, and we believe the approach can be further developed to be more broadly applicable.