PlanLLM: Video Procedure Planning with Refinable Large Language Models

TL;DR

This work addresses video procedure planning from start to goal frames under weak supervision. It introduces PlanLLM, a cross-modal framework that combines a trainable LLM with a Mutual Information Maximization module to fuse sample-specific visual cues with world-level step descriptions. A two-stage progressive training regime enables open-vocabulary step generation while preserving strong closed-set decoding through a refined step decoder and knowledge fusion. Empirical results on CrossTask, NIV, and COIN demonstrate state-of-the-art performance and improved cross-dataset generalization, underscoring the practical value of open-vocabulary planning in embodied AI.

Abstract

Video procedure planning, i.e., planning a sequence of action steps given the video frames of start and goal states, is an essential ability for embodied AI. Recent works utilize Large Language Models (LLMs) to generate enriched action step description texts to guide action step decoding. Although LLMs are introduced, these methods decode the action steps into a closed-set of one-hot vectors, limiting the model's capability of generalizing to new steps or tasks. Additionally, fixed action step descriptions based on world-level commonsense may contain noise in specific instances of visual states. In this paper, we propose PlanLLM, a cross-modal joint learning framework with LLMs for video procedure planning. We propose an LLM-Enhanced Planning module which fully uses the generalization ability of LLMs to produce free-form planning output and to enhance action step decoding. We also propose Mutual Information Maximization module to connect world-level commonsense of step descriptions and sample-specific information of visual states, enabling LLMs to employ the reasoning ability to generate step sequences. With the assistance of LLMs, our method can both closed-set and open vocabulary procedure planning tasks. Our PlanLLM achieves superior performance on three benchmarks, demonstrating the effectiveness of our designs.

Figures (2)

• Figure 1: Language As Supervision methods use a frozen LLM to enhance the textual description of step labels, and are limited to closed-set one-hot vector predictions. In contrast, our Cross-Modal Joint Learning framework utilizes a trainable LLM that allows our model to output both one-hot vectors and free-form open-vocabulary step descriptions, providing stronger generalization for predicting new steps in new datasets.
• Figure 2: The framework of our PlanLLM. PlanLLM mainly consists of three parts: Feature Extraction, Mutual Information Maximization and LLM Enhanced Planning.