ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions

TL;DR

This work tackles generating step-by-step visual instructions grounded in a user’s environment by combining large-scale automatic data collection from instructional videos with a latent video diffusion model conditioned on an input image and per-step prompts. It introduces the ShowHowTo dataset (≈0.6M sequences, 4.5M image-text pairs across 25K tasks) via a four-stage extraction pipeline (transcription, filtering, step extraction, frame alignment) and trains a variable-length, per-step conditioned diffusion model. Evaluations across Step Faithfulness, Scene Consistency, and Task Faithfulness show state-of-the-art performance, supported by quantitative metrics, FID, and user studies, with strong generalization in zero-shot settings. The work enables realistic, scene-aware visual guidance for everyday tasks and has implications for assistive technologies and robot planning.

Abstract

The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image that provides the scene context and the sequence of textual instructions. This is a challenging problem as it requires generating multi-step image sequences to achieve a complex goal while being grounded in a specific environment. Part of the challenge stems from the lack of large-scale training data for this problem. The contribution of this work is thus three-fold. First, we introduce an automatic approach for collecting large step-by-step visual instruction training data from instructional videos. We apply this approach to one million videos and create a large-scale, high-quality dataset of 0.6M sequences of image-text pairs. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image. Third, we evaluate the generated image sequences across three dimensions of accuracy (step, scene, and task) and show our model achieves state-of-the-art results on all of them. Our code, dataset, and trained models are publicly available.

Figures (20)

• Figure 1: Given an input image (left) and ordered step-by-step textual instructions for a task (top), ShowHowTo generates an image sequence of visual instructions. Rows 1 and 2 demonstrate the generation of visual instructions for two recipes starting from the same input image. Rows 2 and 3 show the generation of visual instructions for the same recipe but conditioned on different input images. ShowHowTo generates scene-consistent (e.g., consistency in the person and cutting board) and temporally consistent image sequences (e.g., the bowl of tortilla chips or plate of chicken skewers) that faithfully capture the instructions (e.g., cutting, frying, brushing, adding etc.).
• Figure 2: Our automatic approach for creating the ShowHowTo dataset---a large-scale instructional dataset consisting of step-by-step instruction sequences of image-text pairs to perform diverse HowTo tasks. Examples of step-by-step textual instructions and the corresponding frames are highlighted in green.
• Figure 3: Model architecture. Given an input frame $I_0$ (left) and a variable number of text instructions $\tau_i$ describing each step, our diffusion model generates visual instructions $\hat{I}_i$ that correctly follow the prompts $\tau_i$ and are consistent with the input image $I_0$.
• Figure 4: User study results. Win rates of the ShowHowTo method against baselines from pairwise forced decision user evaluations, divided into step, scene, and task. Values larger than 50% indicate ShowHowTo is preferred over the other methods (right).
• Figure 5: Qualitative comparison using the input image (left) and the textual instructions (top) for the task of making a calzone. The images from the source video are shown in the first row. Except for ShowHowTo, methods either struggle to preserve the input scene or to produce coherent steps.