Using Left and Right Brains Together: Towards Vision and Language Planning

TL;DR

The paper addresses the gap in current large multi-modal models that plan primarily in the language domain by introducing Visual-Language Planning (VLP), which performs concurrent vision planning via video generation and language planning via LLMs before final decision-making. By parallelizing both planning streams and integrating them through a dedicated decision maker, VLP enhances context-aware task execution across vision-language, vision-only, and language-only tasks. The approach is validated on open-domain video QA, video captioning, autonomous driving, and robotics datasets, with ablations showing the contributions of vision planning, language planning, and the decision-making module. This work suggests a path toward more human-like, multi-modal reasoning, while also highlighting the importance of video-generation quality in planning effectiveness.

Abstract

Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning for tasks with inputs of any form. Our framework incorporates visual planning to capture intricate environmental details, while language planning enhances the logical coherence of the overall system. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks. The results demonstrate the superior performance of our approach, indicating that the integration of visual and language planning yields better contextually aware task execution.

Figures (13)

• Figure 1: Existing Large Multi-modality Models (LMMs) directly make the decision based on text and image inputs. Our Vision-Language Planning (VLP) framework conducts both language planning and vision planning first, which serves as the left hemisphere and the right hemisphere of a human brain, and then use a LMM for the final decision making.
• Figure 2: Vision-Language Planning (VLP) Framework. We begin by transforming the user queries into the vision input $I_O$ and language input $T_{O}$ for tasks of different modalities. Subsequently, the vision planning and language planning are conducted in parallel to obtain the vision plan $I_{VP}$ and language plan $T_{LP}$. A decision maker then synthesizes these plans to generate the final output.
• Figure 3: VLP with GPT4-V on vision-related tasks. Languege planning helps the GPT4-V to consider the information from different perspectives and reason sequentially, and vision planning provides the possible future states which are essential for the ultimate decisions.
• Figure 4: VLP with GPT4-V on language-only tasks. GPT4-V gives more detailed and vivid descriptions using the generated videos.
• Figure 5: Decision maker using open-sourced LMM like LLAVA and GPT4-V.