Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding

Yuling Shi, Chaoxiang Xie, Zhensu Sun, Yeheng Chen, Chenxu Zhang, Longfei Yun, Chengcheng Wan, Hongyu Zhang, David Lo, Xiaodong Gu

TL;DR

CodeOCR investigates whether rendering source code as images enables vision-language models to understand code more efficiently. The study systematically evaluates seven multimodal LLMs on four code-understanding tasks across compression ratios from 1x to 8x, with rendering variants including plain, bold, and syntax highlighting. Key findings show that code images can match or surpass text inputs on several tasks, with robust performance especially for Gemini-3 family; visual enhancements help at moderate compression, while information degradation follows a token-line-block hierarchy. The work introduces CodeOCR, a middleware tool for rendering code into configurable images, enabling token-efficient code understanding and paving the way for adaptive, visually-based code representations.

Abstract

Large Language Models (LLMs) have achieved remarkable success in source code understanding, yet as software systems grow in scale, computational efficiency has become a critical bottleneck. Currently, these models rely on a text-based paradigm that treats source code as a linear sequence of tokens, which leads to a linear increase in context length and associated computational costs. The rapid advancement of Multimodal LLMs (MLLMs) introduces an opportunity to optimize efficiency by representing source code as rendered images. Unlike text, which is difficult to compress without losing semantic meaning, the image modality is inherently suitable for compression. By adjusting resolution, images can be scaled to a fraction of their original token cost while remaining recognizable to vision-capable models. To explore the feasibility of this approach, we conduct the first systematic study on the effectiveness of MLLMs for code understanding. Our experiments reveal that: (1) MLLMs can effectively understand code with substantial token reduction, achieving up to 8x compression; (2) MLLMs can effectively leverage visual cues such as syntax highlighting, improving code completion performance under 4x compression; and (3) Code-understanding tasks like clone detection exhibit exceptional resilience to visual compression, with some compression ratios even slightly outperforming raw text inputs. Our findings highlight both the potential and current limitations of MLLMs in code understanding, which points out a shift toward image-modality code representation as a pathway to more efficient inference.

CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding

TL;DR

CodeOCR investigates whether rendering source code as images enables vision-language models to understand code more efficiently. The study systematically evaluates seven multimodal LLMs on four code-understanding tasks across compression ratios from 1x to 8x, with rendering variants including plain, bold, and syntax highlighting. Key findings show that code images can match or surpass text inputs on several tasks, with robust performance especially for Gemini-3 family; visual enhancements help at moderate compression, while information degradation follows a token-line-block hierarchy. The work introduces CodeOCR, a middleware tool for rendering code into configurable images, enabling token-efficient code understanding and paving the way for adaptive, visually-based code representations.

Abstract

Large Language Models (LLMs) have achieved remarkable success in source code understanding, yet as software systems grow in scale, computational efficiency has become a critical bottleneck. Currently, these models rely on a text-based paradigm that treats source code as a linear sequence of tokens, which leads to a linear increase in context length and associated computational costs. The rapid advancement of Multimodal LLMs (MLLMs) introduces an opportunity to optimize efficiency by representing source code as rendered images. Unlike text, which is difficult to compress without losing semantic meaning, the image modality is inherently suitable for compression. By adjusting resolution, images can be scaled to a fraction of their original token cost while remaining recognizable to vision-capable models. To explore the feasibility of this approach, we conduct the first systematic study on the effectiveness of MLLMs for code understanding. Our experiments reveal that: (1) MLLMs can effectively understand code with substantial token reduction, achieving up to 8x compression; (2) MLLMs can effectively leverage visual cues such as syntax highlighting, improving code completion performance under 4x compression; and (3) Code-understanding tasks like clone detection exhibit exceptional resilience to visual compression, with some compression ratios even slightly outperforming raw text inputs. Our findings highlight both the potential and current limitations of MLLMs in code understanding, which points out a shift toward image-modality code representation as a pathway to more efficient inference.
Paper Structure (28 sections, 3 equations, 8 figures, 5 tables)

This paper contains 28 sections, 3 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Experimental Setting
  4. Benchmark and Metrics
  5. Studied Large Language Models
  6. Visual Rendering of Source Code
  7. Baselines and Input Design
  8. Implementation Details
  9. Results and Analysis
  10. RQ1: How Effective are LLMs in Understanding Visualized Code vs. Textual Code?
  11. Feasibility of Code Image Understanding
  12. Model-Specific Variation
  13. RQ2: How Resilient are LLMs to Visual Compression Across Different Coding Tasks?
  14. Compression Effects Across Tasks
  15. Compression Effects Across Models
  16. ...and 13 more sections

Figures (8)

  • Figure 1: An Example of Code Representations across Different Modalities.
  • Figure 2: Multimodal Processing Pipeline for Visualized Code Understanding in MLLMs.
  • Figure 3: Overview of the Empirical Study Design and Core Findings.
  • Figure 4: Examples of Visual Rendering Strategies: Plain, Bold, and Highlight.
  • Figure 5: Performance under Varying Remaining Tokens across Different Tasks.
  • ...and 3 more figures