Context Cascade Compression: Exploring the Upper Limits of Text Compression

TL;DR

This paper tackles the challenge of processing ultra-long contexts in LLMs by proposing Context Cascade Compression (C3), a pure-text, two-LLM cascade that compresses long text into a fixed-length latent context using a small encoder and reconstructs the original text with a larger decoder. At 20× and 40× compression, C3 maintains high reconstruction fidelity (≈98% and ≈93%, respectively), substantially outperforming optical compression approaches like DeepSeek-OCR, which struggle at higher compression ratios. The authors provide a detailed architectural design, training setup, and extensive quantitative and qualitative evaluations on OCR data (Fox benchmark), revealing a sequential information loss pattern analogous to human memory decay rather than uniform degradation. These findings suggest a viable upper bound for text-to-visual compression and a practical path toward ultra-long-context processing, with potential extensions to latent autoencoders for next-generation generative models.

Abstract

Million-level token inputs in long-context tasks pose significant computational and memory challenges for Large Language Models (LLMs). Recently, DeepSeek-OCR conducted research into the feasibility of Contexts Optical Compression and achieved preliminary results. Inspired by this, we introduce Context Cascade Compression C3 to explore the upper limits of text compression. Our method cascades two LLMs of different sizes to handle the compression and decoding tasks. Specifically, a small LLM, acting as the first stage, performs text compression by condensing a long context into a set of latent tokens (e.g., 32 or 64 in length), achieving a high ratio of text tokens to latent tokens. A large LLM, as the second stage, then executes the decoding task on this compressed context. Experiments show that at a 20x compression ratio (where the number of text tokens is 20 times the number of latent tokens), our model achieves 98% decoding accuracy, compared to approximately 60% for DeepSeek-OCR. When we further increase the compression ratio to 40x, the accuracy is maintained at around 93%. This indicates that in the domain of context compression, C3 Compression demonstrates superior performance and feasibility over optical character compression. C3 uses a simpler, pure-text pipeline that ignores factors like layout, color, and information loss from a visual encoder. This also suggests a potential upper bound for compression ratios in future work on optical character compression, OCR, and related fields. Codes and model weights are publicly accessible at https://github.com/liufanfanlff/C3-Context-Cascade-Compression

Figures (5)

• Figure 1: Reconstruction precision and compression ratio of C3 versus Deepseek-OCR on the Fox benchmark. Bars represent precision (left axis), while lines indicate the compression ratio (right axis). C3 (solid bars) demonstrates significantly higher precision than Deepseek-OCR (striped bars) across all tested latent token counts (32, 64, and 100).
• Figure 2: Conceptual comparison of context compression pipelines. The figure contrasts three methods: (1) Original: The baseline tokenization of plain text, leading to a large token count ( 1.2k). (2) Contexts Optical Compression: An indirect method where text is converted to an image and encoded by a ViT, achieving 10x compression into 128 latent tokens. (3) C3 Compression: Our proposed direct method, where a small LLM compresses the text into a minimal set of 32 latent tokens, achieving 40x compression.
• Figure 3: An overview of the C3, which utilizes a cascaded two-LLM design. A smaller encoder LLM compresses a variable-length Input Long Context into a fixed-length Latent Context guided by learnable Context Query tokens. Subsequently, a larger decoder LLM uses this compact Latent Context and a prompt to perform the downstream task, such as reconstructing the original text.
• Figure 4: Qualitative results of text reconstruction using C3 at an extreme compression level (32 latent tokens). Each panel displays the original long text on the left and the model's reconstructed output on the right. The examples showcase the model's high-fidelity performance across diverse scenarios, including: (a) standard English prose, (b) classical Chinese, (c) English text containing non-semantic random characters, and (d) structurally disordered Chinese text. The near-perfect reconstruction in all cases highlights C3's capability for near-lossless compression. Furthermore, we present an analysis of failure cases that occur under extreme compression ratios. (e) A key observation is that in these "bad cases", reconstruction errors tend to be concentrated in the latter half of the original text.
• Figure 5: An analogy of information loss patterns. This figure contrasts two failure modes. Optical compression (middle) leads to a uniform 'blurring' of the entire context. In contrast, C3's information loss is sequential (bottom), fading from the end, which is analogous to the process of human memory decay (top).