Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

TL;DR

Render-of-Thought (RoT) addresses the high computational cost and lack of supervision for intermediate steps in explicit Chain-of-Thought prompting by rendering textual CoT as images and grounding LLM latent states in frozen vision encoders. The method uses a two-stage training procedure—Stage I visual alignment and Stage II latent supervised fine-tuning—to enable autoregressive latent reasoning within a visual latent space, achieving a token reduction of about $3-4x$ and notable inference acceleration. RoT demonstrates competitive performance on mathematical and logical benchmarks across multiple Vision-Language Models, while providing analyzability through visualized latent reasoning trajectories. This approach offers practical speedups and diagnostic capabilities for scalable reasoning in multimodal LLM systems, with potential applicability to broader reasoning tasks and multilingual settings.

Abstract

Chain-of-Thought (CoT) prompting has achieved remarkable success in unlocking the reasoning capabilities of Large Language Models (LLMs). Although CoT prompting enhances reasoning, its verbosity imposes substantial computational overhead. Recent works often focus exclusively on outcome alignment and lack supervision on the intermediate reasoning process. These deficiencies obscure the analyzability of the latent reasoning chain. To address these challenges, we introduce Render-of-Thought (RoT), the first framework to reify the reasoning chain by rendering textual steps into images, making the latent rationale explicit and traceable. Specifically, we leverage the vision encoders of existing Vision Language Models (VLMs) as semantic anchors to align the vision embeddings with the textual space. This design ensures plug-and-play implementation without incurring additional pre-training overhead. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that our method achieves 3-4x token compression and substantial inference acceleration compared to explicit CoT. Furthermore, it maintains competitive performance against other methods, validating the feasibility of this paradigm. Our code is available at https://github.com/TencentBAC/RoT

Figures (9)

• Figure 1: Comparison of Reasoning Paradigms and Efficiency Analysis. (a) Explicit CoT relies on verbose textual generation. (b) Implicit CoT compresses reasoning into latent space. (c) Render-of-Thought utilizes visual rendering as semantic anchors to structure the latent reasoning process.
• Figure 2: Overview of the Render-of-Thought. (a) Rendering Method transforms textual reasoning steps into compact single-line images. (b) Latent Reasoning Method aligns LLM-generated hidden states with visual features via a projection head, enabling the model to perform continuous reasoning within the visual latent space.
• Figure 3: Two-Stage Training Framework. Stage I optimizes the projection head to map linguistic states to visual embeddings while freezing the backbone. Stage II fine-tunes the LLM to autoregressively generate the latent reasoning chain followed by the final answer.
• Figure 4: Inference Time Comparison. We evaluate the average inference time (seconds per sample) on GSM8k-Aug and GSM-Hard datasets using Qwen3-4B-Instruct/Qwen3-VL-4B-Instruct.
• Figure 5: Impact of Rendering Strategies on Training Convergence. The improved single-line rendering demonstrates superior stability and speed compared to the fixed-size square approach.