Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization

TL;DR

This work tackles the challenge of visualizing narratives by exploiting linguistic structure and external knowledge. It introduces VLC-StoryGAN, combining a Memory-Augmented Recurrent Tree Transformer (MARTT) for constituency-tree encoding, ConceptNet-based commonsense graphs, and a dense-captioning based dual task, all trained with an intra-story contrastive loss. Empirical results on PororoSV and FlintstonesSV show strong gains in visual quality, consistency, and relevance, with ablations confirming the importance of each component. The approach highlights the value of structured input and multi-task feedback for coherent, cross-modal story generation with potential for broader storytelling applications.

Abstract

While much research has been done in text-to-image synthesis, little work has been done to explore the usage of linguistic structure of the input text. Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story). Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance. In this paper, we first explore the use of constituency parse trees using a Transformer-based recurrent architecture for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we also incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images within a dual learning setup. We show that off-the-shelf dense-captioning models trained on Visual Genome can improve the spatial structure of images from a different target domain without needing fine-tuning. We train the model end-to-end using intra-story contrastive loss (between words and image sub-regions) and show significant improvements in several metrics (and human evaluation) for multiple datasets. Finally, we provide an analysis of the linguistic and visuo-spatial information. Code and data: https://github.com/adymaharana/VLCStoryGan.

Figures (9)

• Figure 1: Example of generated images from our model VLC-StoryGAN and Duco-StoryGAN maharana2021ducostorygan for the (a) PororoSV and (b) FlintstonesSV datasets.
• Figure 2: Illustration of VLC-StoryGAN architecture. The story encoder is composed of MARTT for encoding sequence of constituency parse trees, and Graph Transformer for encoding commonsense knowledge graphs. The intra-story contrastive loss optimizes semantic alignment while dense captioning loss provides visuo-spatial and semantic feedback about object/characters in generated images.
• Figure 3: Upward cumulative avg. of node embeddings.
• Figure 4: (a) Comparison of predictions from DuCo-StoryGAN and VLC-StoryGAN. (b) The constituency parse for caption of the fourth frame in (a).
• Figure 5: Example of commonsense knowledge for a given caption and its relevance to target image.