PDPP: Projected Diffusion for Procedure Planning in Instructional Videos

TL;DR

This work treats goal-directed procedure planning in instructional videos as learning the joint distribution over entire action sequences conditioned on start/goal observations and a task label. It introduces PDPP, a projected diffusion framework that directly samples plausible action plans, mitigating autoregressive error accumulation and enabling horizon-agnostic, joint training. The approach demonstrates state-of-the-art performance across CrossTask, NIV, COIN, and the VPA task, while reducing supervision requirements by relying on task labels rather than intermediate states or language annotations. The results underscore the effectiveness of conditional diffusion for modeling planning uncertainty and provide practical gains in flexibility and efficiency through condition projection, two-stage prediction, and MoE-based horizon handling. Overall, PDPP advances open-set, multi-horizon procedure planning with strong generalization across datasets and task descriptions.

Abstract

In this paper, we study the problem of procedure planning in instructional videos, which aims to make a plan (i.e. a sequence of actions) given the current visual observation and the desired goal. Previous works cast this as a sequence modeling problem and leverage either intermediate visual observations or language instructions as supervision to make autoregressive planning, resulting in complex learning schemes and expensive annotation costs. To avoid intermediate supervision annotation and error accumulation caused by planning autoregressively, we propose a diffusion-based framework, coined as PDPP, to directly model the whole action sequence distribution with task label as supervision instead. Our core idea is to treat procedure planning as a distribution fitting problem under the given observations, thus transform the planning problem to a sampling process from this distribution during inference. The diffusion-based modeling approach also effectively addresses the uncertainty issue in procedure planning. Based on PDPP, we further apply joint training to our framework to generate plans with varying horizon lengths using a single model and reduce the number of training parameters required. We instantiate our PDPP with three popular diffusion models and investigate a series of condition-introducing methods in our framework, including condition embeddings, MoEs, two-stage prediction and Classifier-Free Guidance strategy. Finally, we apply our PDPP to the Visual Planners for human Assistance problem which requires the goal specified in natural language rather than visual observation. We conduct experiments on challenging datasets of different scales and our PDPP model achieves the state-of-the-art performance on multiple metrics, even compared with those strongly-supervised counterparts. These results further demonstrates the effectiveness and generalization ability of our model.

Figures (7)

• Figure 1: Planning example."Seen observations" denotes the visual inputs for model, "Intermediate supervision" denotes additional supervision used in previous approaches, and "Predicted action" means the actions to be predicted by model. For procedure planning, given a start observation $o_{start}$ and a goal state $o_{goal}$, the model is required to generate an action sequence that can transform $o_{start}$ to $o_{goal}$. Previous approaches rely on heavy intermediate supervision during training, while our model only needs the task class labels (shown in bottom row). We also evaluate another similar Visual Planners for human Assistance (VPA) problem, which takes visual inputs and a language-described goal to make plans. We remove the requirement of visual history and plan directly with the given start observation and goal description.
• Figure 2: Overview of PDPP (horizon $T$ = 3). For procedure planning, we first train a task classifier to generate task condition, which will be used as guidance along with the given observations $o_s$, $o_g$ and horizon condition. Then we compute the denoising process iteratively. For VPA, task label is provided and $o_g$ is set as zero. In each step, we first conduct a condition projection to the input, then predict the initial distribution by the learned model $f_\theta$. After that we calculate $\hat{x}_{n-1}$ with the predicted $\hat{x}_{0}$. We finally select the action dimensions as our result after $N$ denoising steps.
• Figure 3: Schematic diagram for forward and reverse diffusion processes.
• Figure 4: Results for different model backbones with varied task adding methods on three datasets. HowTo100M features are applied to CrossTask.
• Figure 5: Visualization of task, action and intermediate errors.