Glyph: Scaling Context Windows via Visual-Text Compression

TL;DR

Glyph addresses the prohibitive costs of scaling long-context LLMs by rendering ultra-long texts as compact images and processing them with vision-language models. It introduces a three-stage framework—continual pre-training, LLM-driven rendering search, and post-training with SFT/RL and OCR alignment—that discovers rendering configurations balancing compression and accuracy. The method achieves 3–4× token compression with competitive performance to large text-only LLMs, plus substantial efficiency gains and potential for extreme compression to handle 1M-token tasks with 128K-context VLMs. This work proposes a promising orthogonal paradigm to attention-based scaling and demonstrates practical benefits in memory, computation, and cross-modal document understanding.

Abstract

Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-language models (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph.

Figures (6)

• Figure 1: (Upper) Comparison of two paradigms for long-context tasks: conventional approaches directly feeding plain text into LLMs, and the proposed VLM-based paradigm, Glyph, which renders text as compact images to achieve substantial input-token compression. (Lower) Glyph attains competitive performance on LongBench and MRCR, while offering significant compression and inference speedup over its text backbone model on 128K-token inputs.
• Figure 2: Glyph consists of three main stages: continual pre-training on rendered long-text data, LLM-driven genetic search for optimal rendering configurations, and post-training with SFT, RL. Together, these stages enable efficient long-context modeling with visual-text compression.
• Figure 3: Performance comparison of Glyph and the baseline across different context windows, demonstrating that Glyph achieves performance equivalent to longer contexts with substantially shorter context windows.
• Figure 4: Speedup ratios of Glyph over the text backbone model for prefill, decoding, and training across different sequence lengths.
• Figure 5: Model performance degradation across different sequence lengths on the Ruler benchmark.