LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

TL;DR

This work argues that language descriptions offer a more distinctive representation in the latent space for procedure planning, and introduces Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning.

Abstract

Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning.

Figures (7)

• Figure 1: Orange boxes show the labels of actions. Blue boxes show the language descriptions of actions. Left: Visualisation of extracted features of visual observations in the latent space. Each point represents one video. Middle: Visualisation of text embeddings of actions in the latent space. Right: Start visual observation of "Fill filter with ground coffee" and "Even the surface of coffee".
• Figure 2: Overview of our proposed LAP. Professor Forcing: We finetune a VLM using professor forcing to predict the elaborated description $\hat{D}_{a_i}$ of an action $a_i$. The elaborated description is obtained by prompting $a_i$ to a pretrained LLM. Video-to-Text: we use the finetuned VLM to generate descriptions for start and goal visual observation $V_{a_s}$ and $V_{a_g}$ and find out the descriptions that are most similar to $D_{a_s}$ and $D_{a_g}$ to represent the predicted action $\hat{a}_s$ and $\hat{a}_g$. Then the extractor extracts the corresponding embeddings $E_{\hat{a}_s}$ and $E_{\hat{a}_g}$. Planning: We use a diffusion model to plan the action sequence. We use the embeddings of start and end embeddings $E_{\hat{a}_s}$ and $E_{\hat{a}_g}$ as well as action sequence to construct the input to the diffusion model. During training, only the action dimension is noised.
• Figure 3: VLM finetuning: we use professor forcing and scheduled sampling to finetune the VLM. During the forward pass, the VLM either use teaching forcing or free running to generate tokens with a scheduled probability. A discriminator is used to minimise the distribution distances between them so that the generated tokens are as close to ground truth tokens.
• Figure 4: Visualisation of features of visual observation and text embedding in latent space on CrossTask, Coin and NIV dataset. The first row shows the visual observation and the second row shows the text embedding. All visualisations are the samples from test sets with time horizon $T=3$.
• Figure 5: Qualitative results on NIV dataset. Purple boxes indicate the ground truth actions. Green boxes indicate the correctly predicted actions in action sequences. Red boxes indicate the wrong predictions. Top: predictions when $\hat{a}_s$ and $\hat{a}_g$ are "unknown" from VLM. Middle: predictions when $\hat{a}_s$ is "unknown" and $\hat{a}_g$ is correct. Bottom: predictions when $\hat{a}_s$ is correct and $\hat{a}_g$ is "unknown".