Visual Planning: Let's Think Only with Images

TL;DR

Visual Planning tackles the modality gap in reasoning for vision-first tasks by enabling planning entirely in the visual domain. It introduces VPRL, a two-stage reinforcement learning framework powered by GRPO, to train a large vision system to generate sequences of intermediate images that encode planning steps. Across FrozenLake, Maze, and MiniBehavior, Visual Planning with VPRL outperforms text-based planning and VPFT, with notable improvements in Exact Match and generalization to unseen layouts. The work suggests visual, language-free planning as a practical complement to language-based reasoning for spatial and dynamic tasks.

Abstract

Recent advancements in Large Language Models (LLMs) and their multimodal extensions (MLLMs) have substantially enhanced machine reasoning across diverse tasks. However, these models predominantly rely on pure text as the medium for both expressing and structuring reasoning, even when visual information is present. In this work, we argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information. Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations for these "vision-first" tasks, as a supplementary channel to language-based reasoning. In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions. We introduce a novel reinforcement learning framework, Visual Planning via Reinforcement Learning (VPRL), empowered by GRPO for post-training large vision models, leading to substantial improvements in planning in a selection of representative visual navigation tasks, FrozenLake, Maze, and MiniBehavior. Our visual planning paradigm outperforms all other planning variants that conduct reasoning in the text-only space. Our results establish Visual Planning as a viable and promising supplement to language-based reasoning, opening new avenues for tasks that benefit from intuitive, image-based inference.

Figures (15)

• Figure 1: Comparison of reasoning paradigms. The traditional approaches (top and middle rows) generate verbose and inaccurate textual plan, while the Visual Planning paradigm (bottom row) predicts the next visual state directly, forming a pure image trajectory without language mediation.
• Figure 2: An overview of the proposed VPRL framework, illustrated with autoregressive large vision models for image generation in the context of a visual navigation task. We train the visual policy model with GRPO, using the progress reward that encourages progressing actions and penalizes invalid actions, yielding goal-aligned visual planning.
• Figure 3: Illustration of each task with generated visual planning traces from LVM, covering different types of actions (optimal, non-optimal and invalid). More cases can be found in Appendix \ref{['appsubsec:examples']}.
• Figure 4: Visualization of a test example from FrozenLake comparing visual planning variants (VPFT and VPRL) with language-based reasoning variants.
• Figure 5: Evaluation of model performance on FrozenLake under varying levels of difficulty. As the environment complexity increases with larger grid sizes, language-based reasoning methods experience a sharp decline in performance, whereas visual planning methods exhibit a more gradual drop, demonstrating greater robustness.