Adaptive Visual Conditioning for Semantic Consistency in Diffusion-Based Story Continuation

TL;DR

This work tackles semantic drift in diffusion-based story continuation by introducing Adaptive Visual Conditioning (AVC), which adaptively leverages prior visual context based on semantic alignment with the current narration. It combines CLIP-based semantic memory selection to identify the most relevant past frame with a progressive conditioning strategy that restricts visual guidance to early diffusion steps when relevance is low. To strengthen textual supervision, the authors re-caption StorySalon images using large language models, improving alignment between text and visuals. Quantitative results and human evaluations show that AVC enhances coherence, semantic consistency, and perceptual quality, achieving state-of-the-art performance without additional training and demonstrating robustness in challenging, conflicting-context scenarios.

Abstract

Story continuation focuses on generating the next image in a narrative sequence so that it remains coherent with both the ongoing text description and the previously observed images. A central challenge in this setting lies in utilizing prior visual context effectively, while ensuring semantic alignment with the current textual input. In this work, we introduce AVC (Adaptive Visual Conditioning), a framework for diffusion-based story continuation. AVC employs the CLIP model to retrieve the most semantically aligned image from previous frames. Crucially, when no sufficiently relevant image is found, AVC adaptively restricts the influence of prior visuals to only the early stages of the diffusion process. This enables the model to exploit visual context when beneficial, while avoiding the injection of misleading or irrelevant information. Furthermore, we improve data quality by re-captioning a noisy dataset using large language models, thereby strengthening textual supervision and semantic alignment. Quantitative results and human evaluations demonstrate that AVC achieves superior coherence, semantic consistency, and visual fidelity compared to strong baselines, particularly in challenging cases where prior visuals conflict with the current input.

Figures (6)

• Figure 1: Distributions of CLIP similarity scores for first caption, BLIP, BLIP-2 and GPT captions on the StorySalon test set.
• Figure 2: Overview of our proposed Adaptive Visual Conditioning (AVC) framework. Given the current text description and the most semantically relevant previous frame (selected by CLIP-based similarity), both are encoded into the latent space. The image latent is perturbed through the diffusion process, while the text embedding provides semantic guidance. Depending on the similarity score $s$, Image conditioning is injected adaptively up to timestep $m(s)$, after which only the text condition remains: (i) for low $s$, image guidance is applied only in early steps, (ii) for medium $s$, it is gradually extended, and (iii) for high $s$, both text and image are used throughout the full denoising process. This adaptive design balances reliance on textual and visual information based on the reliability of the retrieved frame.
• Figure 3: Qualitative examples of re-captioning. The new captions lead to semantically richer descriptions and more visually coherent generations compared to the previous captions.
• Figure 4: Examples of CLIP-based best image selection. Compared to the previous selection, the chosen images better align with the textual descriptions under new captions.
• Figure 5: Qualitative examples of Adaptive Visual Conditioning (AVC). AVC enhances alignment between captions and generated images, with improved text consistency and reduced visual artifacts compared to fixed-timestep conditioning.