Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs

TL;DR

Latent Sketchpad tackles the challenge of complex multimodal reasoning in MLLMs by enabling internal visual thinking. It adds a Context-Aware Vision Head to autoregressively generate visual latents during reasoning and a pretrained Sketch Decoder to render these latents into interpretable sketches, all while preserving the backbone's reasoning capabilities. Evaluations on the MazePlanning dataset show that this approach yields comparable or superior reasoning performance and provides usable visual traces across frontier backbones like Gemma3 and Qwen2.5-VL, with GPT-4o further benefiting from the added visual reasoning. The work demonstrates broad plug-and-play applicability and opens avenues for richer, more interpretable human–AI interactions in multimodal tasks.

Abstract

While Multimodal Large Language Models (MLLMs) excel at visual understanding, they often struggle in complex scenarios that require visual planning and imagination. Inspired by how humans use sketching as a form of visual thinking to develop and communicate ideas, we introduce Latent Sketchpad, a framework that equips MLLMs with an internal visual scratchpad. The internal visual representations of MLLMs have traditionally been confined to perceptual understanding. We repurpose them to support generative visual thought without compromising reasoning ability. Building on frontier MLLMs, our approach integrates visual generation directly into their native autoregressive reasoning process. It allows the model to interleave textual reasoning with the generation of visual latents. These latents guide the internal thought process and can be translated into sketch images for interpretability. To realize this, we introduce two components: a Context-Aware Vision Head autoregressively produces visual representations, and a pretrained Sketch Decoder renders these into human-interpretable images. We evaluate the framework on our new dataset MazePlanning. Experiments across various MLLMs show that Latent Sketchpad delivers comparable or even superior reasoning performance to their backbone. It further generalizes across distinct frontier MLLMs, including Gemma3 and Qwen2.5-VL. By extending model's textual reasoning to visual thinking, our framework opens new opportunities for richer human-computer interaction and broader applications. More details and resources are available on our project page: https://latent-sketchpad.github.io/.

Figures (13)

• Figure 1: (a) Latent Sketchpad extends frontier MLLMs (e.g. Gemma3 and Qwen2.5-VL) to interleave text and visual latents generation, incorporating visual thoughts into reasoning. (b) The framework enables interleaved generation by equipping the pretrained MLLM with a Vision Head to generate visual latents autoregressively. A separately pretrained Sketch Decoder visualizes these latents into interpretable sketches.
• Figure 2: Architecture of the Context-Aware Vision Head and Sketch Decoder. The Vision Head transforms hidden states from the MLLM backbone into visual latents. The Sketch Decoder operates independently, converting these latents into sketch-style images for visualization and interpretability.
• Figure 3: Illustration of generalization and compatibility of the pretrained Sketch Decoder. (a) Quantitative reconstruction results (SSIM) across different vision encoders (OpenCLIP, Qwen2.5-VL and Gemma3) on unseen samples from MazePlanning. (b) Qualitative examples of reconstructed sketches from visual latents produced by each encoder.
• Figure 4: Qualitative analysis illustrating visualizations from Latent Sketchpad-enhanced Gemma3 and Qwen2.5-VL on in-distribution mazes.
• Figure 5: Visualizations from Latent Sketchpad on Gemma3 and Qwen2.5-VL in the OOD test set.