Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression

TL;DR

This work investigates how to handle ultra-long code contexts using visual compression instead of traditional textual pruning. It introduces LongCodeOCR, which renders code into compact 2D image sequences and feeds them into vision–language models, preserving global structure to support cross-file dependencies. Across four long-context benchmarks, visual compression substantially outperforms textual baselines at comparable compression ratios, especially in global understanding tasks like code summarization, while revealing a coverage–fidelity trade-off: broader context coverage benefits global reasoning, but fidelity bottlenecks can limit symbol-level precision. The results suggest practical advantages for ultra-long code tasks, with implications for interactive developer workflows and new directions in hybrid compression strategies and rendering optimizations.

Abstract

Large Language Models (LLMs) struggle with long-context code due to window limitations. Existing textual code compression methods mitigate this via selective filtering but often disrupt dependency closure, causing semantic fragmentation. To address this, we introduce LongCodeOCR, a visual compression framework that renders code into compressed two-dimensional image sequences for Vision-Language Models (VLMs). By preserving a global view, this approach avoids the dependency breakage inherent in filtering. We systematically evaluate LongCodeOCR against the state-of-the-art LongCodeZip across four benchmarks spanning code summarization, code question answering, and code completion. Our results demonstrate that visual code compression serves as a viable alternative for tasks requiring global understanding. At comparable compression ratios ($\sim$1.7$\times$), LongCodeOCR improves CompScore on Long Module Summarization by 36.85 points over LongCodeZip. At a 1M-token context length with Glyph (a specialized 9B VLM), LongCodeOCR maintains higher accuracy than LongCodeZip while operating at about 4$\times$ higher compression. Moreover, compared with LongCodeZip, LongCodeOCR drastically reduces compression-stage overhead (reducing latency from $\sim$4.3 hours to $\sim$1 minute at 1M tokens). Finally, our results characterize a fundamental coverage--fidelity trade-off: visual code compression retains broader context coverage to support global dependencies, yet faces fidelity bottlenecks on exactness-critical tasks; by contrast, textual code compression preserves symbol-level precision while sacrificing structural coverage.

Figures (2)

• Figure 1: A motivating example from RepoBench-P illustrating how selective filtering can break dependency closure. The SingleCellSelection definition (left, highlighted) is required for correct code completion but is removed after filtering (right, indicated by the masked region), causing irreversible coverage loss and an incorrect completion.
• Figure 2: Framework overview of LongCodeOCR and two paradigms for long-context code compression. Stages 1--3 depict how massive code contexts are rendered into a 2D image sequences and encoded into visual tokens for instruction-conditioned, cross-modal reasoning. The bottom panel highlights the contrast between the global-preserving visual view and the fragmented textual view.