Global Context Compression with Interleaved Vision-Text Transformation

TL;DR

The paper tackles the quadratic cost of self-attention in long-context transformers and the inefficiency of existing partial context compression methods. It introduces Global Context Compression (GCC) through VIST2, a Transformer that interleaves textual chunks with their rendered visual encodings and trains via Optical Language Modeling (OLM) to support both prefilling and iterative inference. A multi-stage training pipeline—image captioning, MT-OCR, OLM, and modal-interleaved instruction tuning—enables effective cross-modal alignment and robust long-context performance. Across a 0.6B–8B model family, VIST2 achieves up to $3\times$ speedup in first-token generation and around $77\%$ memory reduction and $74\%$ FLOPS reduction at a $4\times$ compression ratio, while delivering strong results on long-context benchmarks and maintaining core reasoning capabilities. The work demonstrates practical, scalable efficiency gains for long document understanding and generation with GCC-based architectures.

Abstract

Recent achievements of vision-language models in end-to-end OCR point to a new avenue for low-loss compression of textual information. This motivates earlier works that render the Transformer's input into images for prefilling, which effectively reduces the number of tokens through visual encoding, thereby alleviating the quadratically increased Attention computations. However, this partial compression fails to save computational or memory costs at token-by-token inference. In this paper, we investigate global context compression, which saves tokens at both prefilling and inference stages. Consequently, we propose VIST2, a novel Transformer that interleaves input text chunks alongside their visual encoding, while depending exclusively on visual tokens in the pre-context to predict the next text token distribution. Around this idea, we render text chunks into sketch images and train VIST2 in multiple stages, starting from curriculum-scheduled pretraining for optical language modeling, followed by modal-interleaved instruction tuning. We conduct extensive experiments using VIST2 families scaled from 0.6B to 8B to explore the training recipe and hyperparameters. With a 4$\times$ compression ratio, the resulting models demonstrate significant superiority over baselines on long writing tasks, achieving, on average, a 3$\times$ speedup in first-token generation, 77% reduction in memory usage, and 74% reduction in FLOPS. Our codes and datasets will be public to support further studies.

Figures (7)

• Figure 1: The illustration of context compression of the Transformer. Right arrow indicates transforming a text chunk to its latent representation.
• Figure 2: The illustration of pre-training.
• Figure 3: Monitoring stage-2 pre-training loss of VIST2 models.
• Figure 4: Comparison of efficiency.
• Figure 5: Training loss of image captioning.