Planning with Vision-Language Models and a Use Case in Robot-Assisted Teaching

TL;DR

The paper introduces Image2PDDL, a framework that automates PDDL problem generation by translating visual initial states and goal descriptions into symbolic planning representations via Vision-Language Models. It presents a three-step pipeline that uses structured prompts and predefined planning domains to produce syntactically correct and executable PDDL across Blocksworld, Sliding-Tile, and Kitchen environments, with evaluation on syntax and content accuracy. The work demonstrates strong syntax reliability and domain-specific content challenges, and explores a salient use-case in robot-assisted teaching for ASD within the TEACCH framework, including automated assessment of shoe-box tasks. Overall, Image2PDDL advances accessible, scalable AI planning by tightly integrating perception and symbolic reasoning and lays groundwork for broader real-world applications in education and assistive robotics.

Abstract

Automating the generation of Planning Domain Definition Language (PDDL) with Large Language Model (LLM) opens new research topic in AI planning, particularly for complex real-world tasks. This paper introduces Image2PDDL, a novel framework that leverages Vision-Language Models (VLMs) to automatically convert images of initial states and descriptions of goal states into PDDL problems. By providing a PDDL domain alongside visual inputs, Imasge2PDDL addresses key challenges in bridging perceptual understanding with symbolic planning, reducing the expertise required to create structured problem instances, and improving scalability across tasks of varying complexity. We evaluate the framework on various domains, including standard planning domains like blocksworld and sliding tile puzzles, using datasets with multiple difficulty levels. Performance is assessed on syntax correctness, ensuring grammar and executability, and content correctness, verifying accurate state representation in generated PDDL problems. The proposed approach demonstrates promising results across diverse task complexities, suggesting its potential for broader applications in AI planning. We will discuss a potential use case in robot-assisted teaching of students with Autism Spectrum Disorder.

Figures (7)

• Figure 1: Image2PDDL operates in three main steps. First, the image of the initial state is translated into a predefined state format. Next, either an image or text description of the goal state is similarly converted to the same state format. Finally, both states are used to generate a PDDL problem based on predefined domains and examples.
• Figure 2: Images of example scenarios of each domain.
• Figure 3: Types of Errors in the Kitchen Domain: The left chart represents errors when using an image to describe the goal state, while the right chart represents errors when using a text to describe the goal state.
• Figure 4: Shoebox tasks as a use case for robot-assisted teaching.
• Figure 5: Instance of the distribution task that requires single snapshot to obtain specification from ChatGPT4o.