Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks

TL;DR

This work tackles the challenge of generating coherent image sequences to accompany real-world manual tasks by introducing a Sequential Latent Diffusion Model (SLDM) that couples an LDM with a Sequence Context Decoder to produce contextual captions and a Sequence Conditioned Reverse Diffusion that leverages prior latent vectors for visual coherence. The approach uses a context window for captions generated by InstructBLIP and a latent-vector copying mechanism to initialize each step's diffusion process, guided by CLIP-based similarity to select conditioning seeds. Empirical results show that the proposed method is preferred by human annotators in 46.6% of cases (vs 26.6% for the second-best method) and achieves higher DreamSim scores while maintaining CLIPScore, with promising but partial generalization to unseen DIY tasks. Overall, the method advances coherent visual guidance for complex manual tasks and suggests a scalable path for integrating semantic context with visual generation in real-world scenarios.

Abstract

Multistep instructions, such as recipes and how-to guides, greatly benefit from visual aids, such as a series of images that accompany the instruction steps. While Large Language Models (LLMs) have become adept at generating coherent textual steps, Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences. The most challenging aspect is that each generated image needs to adhere to the relevant textual step instruction, as well as be visually consistent with earlier images in the sequence. To address this problem, we propose an approach for generating consistent image sequences, which integrates a Latent Diffusion Model (LDM) with an LLM to transform the sequence into a caption to maintain the semantic coherence of the sequence. In addition, to maintain the visual coherence of the image sequence, we introduce a copy mechanism to initialise reverse diffusion processes with a latent vector iteration from a previously generated image from a relevant step. Both strategies will condition the reverse diffusion process on the sequence of instruction steps and tie the contents of the current image to previous instruction steps and corresponding images. Experiments show that the proposed approach is preferred by humans in 46.6% of the cases against 26.6% for the second best method. In addition, automatic metrics showed that the proposed method maintains semantic coherence and visual consistency across steps in both domains.

Figures (22)

• Figure 1: The properties of the elements in illustrations should remain coherent throughout the whole sequence.
• Figure 2: Example captions generated by InstructBLIP. In the "No Context" example, the model only receives the image. In the "Current Step as Context" example, the model receives the image plus the "Original Step".
• Figure 3: The proposed method uses the sequence context decoder to maintain semantic coherence. The reverse diffusion process uses a conditioning seed $z^{i}_T$ that is copied from a previous step and iteration $z^{j}_k$. See Equation \ref{['eq:loss_sldm']}.
• Figure 4: Automatic evaluation of image sequence. CLIP-Score CLIPScore measures the alignment between the step and the image. DreamSim dreamsim measures similarity between visual illustrations in the sequence.
• Figure 5: Examples of recipe illustrations with different methods for maintaining visual coherence.