Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models

TL;DR

This work introduces Can-Do, a benchmark dataset designed to evaluate embodied planning abilities through more diverse and complex scenarios than previous datasets, and proposes NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans.

Abstract

Large multimodal models have demonstrated impressive problem-solving abilities in vision and language tasks, and have the potential to encode extensive world knowledge. However, it remains an open challenge for these models to perceive, reason, plan, and act in realistic environments. In this work, we introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities through more diverse and complex scenarios than previous datasets. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. The data encompasses diverse aspects of commonsense knowledge, physical understanding, and safety awareness. Our fine-grained analysis reveals that state-of-the-art models, including GPT-4V, face bottlenecks in visual perception, comprehension, and reasoning abilities. To address these challenges, we propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans. Experimental results demonstrate the effectiveness of our framework compared to strong baselines. Our code and dataset are available at https://embodied-planning.github.io.

Figures (6)

• Figure 1: Examples of embodied planning problems in our Can-Do dataset.
• Figure 2: Data statistics and distribution.
• Figure 3: Comparison of plan lengths of datasets.
• Figure 4: Preliminary study on embodied planning bottlenecks across different planning categories. To investigate visual perception, comprehension, and reasoning abilities, we progressively prompt multimodal models with ground-truth information of the initial state and goal state of the environment.
• Figure 5: An overview of NeuroGround, our neuro-symbolic framework for grounded planning. To enhance visual perception and goal comprehension of the model, we leverage state-grounded planning which explicitly guides the model to perceive and condition on the environment states before starting to generate the plan. To mitigate the plan generation challenges, we leverage a symbolic engine to augment the model-generated plan, based on the perceived states. For brevity, we do not show the task description and demonstration inputs.