Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos

TL;DR

This work addresses long-horizon procedural planning in instructional videos by introducing KEPP, a Knowledge-Enhanced Procedure Planning system that leverages a probabilistic procedural knowledge graph (P$^2$KG) extracted from training procedure plans. KEPP decomposes planning into identifying the initial and final steps from start and goal visuals using a Conditioned Projected Diffusion Model, then retrieves top plan paths from the P$^2$KG to condition a diffusion-based planner for the full sequence. The main contributions are the P$^2$KG framework to capture implicit causal constraints and plan variability with minimal supervision, and a two-stage learning paradigm that integrates graph-derived knowledge with perception-driven steps. Experimental results across CrossTask, COIN, and NIV show state-of-the-art performance, with ablations validating the effectiveness of graph-conditioned planning and highlighting trade-offs with LLM-based planning and graph design. This approach promises scalable, knowledge-grounded planning for real-world instructional tasks and suggests directions for zero-shot extension and improved robustness for start/end-step predictions.

Abstract

In this paper, we explore the capability of an agent to construct a logical sequence of action steps, thereby assembling a strategic procedural plan. This plan is crucial for navigating from an initial visual observation to a target visual outcome, as depicted in real-life instructional videos. Existing works have attained partial success by extensively leveraging various sources of information available in the datasets, such as heavy intermediate visual observations, procedural names, or natural language step-by-step instructions, for features or supervision signals. However, the task remains formidable due to the implicit causal constraints in the sequencing of steps and the variability inherent in multiple feasible plans. To tackle these intricacies that previous efforts have overlooked, we propose to enhance the capabilities of the agent by infusing it with procedural knowledge. This knowledge, sourced from training procedure plans and structured as a directed weighted graph, equips the agent to better navigate the complexities of step sequencing and its potential variations. We coin our approach KEPP, a novel Knowledge-Enhanced Procedure Planning system, which harnesses a probabilistic procedural knowledge graph extracted from training data, effectively acting as a comprehensive textbook for the training domain. Experimental evaluations across three widely-used datasets under settings of varying complexity reveal that KEPP attains superior, state-of-the-art results while requiring only minimal supervision.

Figures (11)

• Figure 1: Expert trajectories bi2021procedure of the 'Make Jello Shots' task from the CrossTask dataset zhukov2019cross. Heavier color indicates more frequently visited path. This depicts the complexities of the procedure planning task, arising from the subtle causal links in step sequencing (e.g., steps like 'stir mixture' or 'pour mixture' typically occur after adding individual ingredients), the varied probabilities of transitioning between steps, and the diversity in plans viable for a given starting point and an intended outcome. Motivated by these nuanced challenges, we propose Knowledge-Enhanced Procedure Planning (KEPP) with the use of a probabilistic procedure knowledge graph to capture and represent these intricacies
• Figure 2: Overview of our methodology. We introduce KEPP, a Knowledge-Enhanced Procedure Planning system for instructional videos, leveraging a Probabilistic Procedural Knowledge Graph (P$^2$KG). KEPP breaks down procedure planning into two parts: predicting initial and final steps from visual states, and crafting a procedure plan based on the procedural knowledge retrieved from P$^2$KG, conditioned on the predicted first and last action steps. KEPP requires minimal annotations and enhances planning effectiveness
• Figure 3: Qualitative analysis of the 'Make Jello Shots' task
• Figure 4: Qualitative analysis of the 'Change a Tire' task
• Figure 5: Example of a sub-graph in our probabilistic procedure knowledge graph (P$^2$KG) for CrossTask dataset. This graph effectively encapsulates real-world knowledge of distinct transition probabilities between steps, e.g., the probability of transitioning from 'start loose' to 'jack up' is 0.65, in contrast to a mere 0.14 for the reverse transition--the P$^2$KG reflects the common real-life practice where loosening the lug nuts before jacking up the car leads to a safer and more efficient tire change.