VisAgent: Narrative-Preserving Story Visualization Framework

TL;DR

VisAgent tackles the challenge of preserving narrative meaning in story visualization by coupling a training-free, multi-agent story-distillation module with a three-agent image-rendering pipeline. The story module converts plain narratives into layered FG/BG prompts through a three-act analysis, with user feedback and reflection to minimize hallucinations. The image module builds semantic-consistent visuals by generating FG/BG elements separately, determining subject placement with a scene locator, and rendering via a SA-CA layer that fuses prompts and latent guidance. Quantitative metrics (FID, CCS, TIS) and human studies show that VisAgent yields higher narrative fidelity and image coherence compared with baselines like AutoStudio and GPT-4o-based prompts, demonstrating practical applicability for narrative-aware visualization.

Abstract

Story visualization is the transformation of narrative elements into image sequences. While existing research has primarily focused on visual contextual coherence, the deeper narrative essence of stories often remains overlooked. This limitation hinders the practical application of these approaches, as generated images frequently fail to capture the intended meaning and nuances of the narrative fully. To address these challenges, we propose VisAgent, a training-free multi-agent framework designed to comprehend and visualize pivotal scenes within a given story. By considering story distillation, semantic consistency, and contextual coherence, VisAgent employs an agentic workflow. In this workflow, multiple specialized agents collaborate to: (i) refine layered prompts based on the narrative structure and (ii) seamlessly integrate \gt{generated} elements, including refined prompts, scene elements, and subject placement, into the final image. The empirically validated effectiveness confirms the framework's suitability for practical story visualization applications.

Figures (6)

• Figure 1: Problem statement of narrative-preserving story visualization and our improvements for overcoming limitations.
• Figure 2: Architectural overview of VisAgent, a multi-agent framework designed for story visualization. The recycle symbol () represents a process that repeats until approval. $\oplus$ refers to the process of generating a GB prompt by concatenating BG and FG prompts to each scene while $\otimes$ specifically denotes a segmentation-based image stitching process.
• Figure 3: Schematic of semantic-aware cross-attention layer.
• Figure 4: Results of qualitative evaluation: narrative-preserving story visualization results of VisAgent using refined layered prompts as listed in Table \ref{['table_qualitative_eval_test']}.
• Figure 5: Results of qualitative evaluation: performance analysis of five example prompts-based story visualizations compared to the baseline.