ARC-Encoder: learning compressed text representations for large language models

TL;DR

ARC-Encoder tackles long-context challenges in large language models by learning an encoder that compresses text into $x$-times fewer continuous representations to feed a frozen decoder. It introduces two pretraining objectives Reconstruction and Continuation and a pooling mechanism in the final attention layer, enabling a decoder-agnostic compressor that can also extend context for long documents. A shared encoder with per-decoder MLP projectors supports multi-decoder adaptation and even new decoders with minimal fine-tuning, enabling decoder-agnostic compression and long-context extension. Empirically, ARC-Encoder achieves state-of-the-art or competitive results on QA, translation, and summarization while delivering inference speedups, and a memory-efficient option for precomputing compressed representations, highlighting practical utility for portable encoders across LLMs.

Abstract

Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a systematic study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x\!\in\!\{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves state-of-the-art performance on several benchmarks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that work seamlessly with multiple LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder , fine-tuning dataset and pretrained models are available at https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .

Figures (6)

• Figure 1: Pooling in ARC$_2$-Encoder. In the last SA module, queries are merged by pairs of successive tokens.
• Figure 2: ARC-Encoder pretraining tasks. The encoder, special tokens and the MLP are trained through two alternating tasks: a) Reconstruction: compressed tokens are given to the decoder which is teacher-forced to replicate the full text tokens; b) Continuation: a subpart of the tokens in the sequence are compressed and the decoder is teacher-forced to continue starting from the partially compressed sequence. Illustration for ARC$_2$-Encoder.
• Figure 3: Extending an ARC-Encoder to a new decoder.
• Figure 4: Compression results with varying MLP dimensional bottlenecks and number of bits per dimension (B/d).
• Figure 5: Score for various pairs of pooling factor between pretraining and fine-tuning.