$I^2G$: Generating Instructional Illustrations via Text-Conditioned Diffusion

TL;DR

The work addresses generating instructional visuals from procedural text by decomposing goals and steps and grounding visuals via a text-conditioned diffusion model. It introduces a pairwise cross-image consistency mechanism, a constituency-based text encoding scheme, and a reward-guided fine-tuning framework that leverages LLM-sampled text. Experiments on HT-Step, CaptainCook4D, and WikiAll show improvements over baselines in text-image alignment and sequential coherence across multiple datasets. The framework advances grounding procedural language in visuals with potential applications in education and multimodal instruction understanding.

Abstract

The effective communication of procedural knowledge remains a significant challenge in natural language processing (NLP), as purely textual instructions often fail to convey complex physical actions and spatial relationships. We address this limitation by proposing a language-driven framework that translates procedural text into coherent visual instructions. Our approach models the linguistic structure of instructional content by decomposing it into goal statements and sequential steps, then conditioning visual generation on these linguistic elements. We introduce three key innovations: (1) a constituency parser-based text encoding mechanism that preserves semantic completeness even with lengthy instructions, (2) a pairwise discourse coherence model that maintains consistency across instruction sequences, and (3) a novel evaluation protocol specifically designed for procedural language-to-image alignment. Our experiments across three instructional datasets (HTStep, CaptainCook4D, and WikiAll) demonstrate that our method significantly outperforms existing baselines in generating visuals that accurately reflect the linguistic content and sequential nature of instructions. This work contributes to the growing body of research on grounding procedural language in visual content, with applications spanning education, task guidance, and multimodal language understanding.

Figures (5)

• Figure 1: The qualitative results of our method, compared with baseline models, are illustrated in the figure. The baseline model struggles to capture the progression in the text, whereas our method successfully captures this progression and achieves a more illustrative result. However, the image lacks in StackDiffusion due to the model's limitation of not generating more than 6 steps.
• Figure 2: We randomly sample pairs $(v_i,s_i)$ and $(v_j,s_j)$, apply a custom adjacency mask to fuse latent representations, and decode them back into images. The constituency parser (Sec. \ref{['sec:enc']}) splits text to handle length constraints.
• Figure 3: We demonstrate that the goal text often serves as contextual information with limited relation to the visual content, and CLIP frequently produces the highest scores across various datasets. Additionally, the MLLM often fails to align with human judgment, providing high scores that do not correspond with the intended goals.
• Figure 4: Compared to goal image alignment, MLLMs perform better in aligning step text with images, showing high agreement with human judgment. Although human evaluators tend to assign a range of scores, they generally award high scores.
• Figure 5: Images of successful and failed generation