VinaBench: Benchmark for Faithful and Consistent Visual Narratives

TL;DR

VinaBench tackles the challenge of faithful and consistent visual narrative generation by augmenting text-image pairs with commonsense grounding and discourse constraints. It introduces a constraint-driven learning and evaluation framework, including novel alignment and VQA-based fine-grained metrics that assess both fidelity to the input narrative and long-range visual consistency. Across multiple datasets and three generative models, incorporating these constraints improves alignment and cohesion, with human evaluations broadly supporting the automatic metrics, though room for improvement remains relative to human-crafted references. The work offers a practical benchmark and evaluation toolkit to advance robust, interpretable visual storytelling and provides a pathway for more reliable downstream Vision-Language applications.

Abstract

Visual narrative generation transforms textual narratives into sequences of images illustrating the content of the text. However, generating visual narratives that are faithful to the input text and self-consistent across generated images remains an open challenge, due to the lack of knowledge constraints used for planning the stories. In this work, we propose a new benchmark, VinaBench, to address this challenge. Our benchmark annotates the underlying commonsense and discourse constraints in visual narrative samples, offering systematic scaffolds for learning the implicit strategies of visual storytelling. Based on the incorporated narrative constraints, we further propose novel metrics to closely evaluate the consistency of generated narrative images and the alignment of generations with the input textual narrative. Our results across three generative vision models demonstrate that learning with VinaBench's knowledge constraints effectively improves the faithfulness and cohesion of generated visual narratives.

Figures (12)

• Figure 1: Overview of VinaBench. We augment existing visual-textual narrative pairs with discourse and commonsense constraints, to promote the learning of consistent and faithful visual narrative generation and its evaluation. The commonsense constraints consist of links that ground the visual entities (extracted from image captions) to their associated textual narrative entities, as labeled by the phrases paired with the same color. The discourse constraints include scene-specific narrative features that trace the dynamics of basic narrative elements, i.e., characters, time and location, and global narrative features that describe static character attributes and image appearance style.
• Figure 2: Overview of VinaBench data construction pipeline. We use hybrid VLMs and LLMs to annotate the discourse features and commonsense links underlying visual-textual narrative pairs.
• Figure 3: Pearson correlation coefficients between human and automatic evaluation metrics on VWP narratives. Alignment and Consistency in automatic evaluation metrics denote the average of our VQA-based fine-grained alignment and consistency metrics, respectively, rooted on MiniCPM-V-2.6.
• Figure 4: Visual narratives generated by MM-Interleaved with and without LLM-generated narrative constraints, and the gold reference.
• Figure S1: Correlation between generated visual narrative images and augmented narrative constraints (either from gold labels or generated by LLM, Llama3.1-70B-Instruct), w.r.t. their CLIP embedding similarity to the input textual narrative. Data samples are from MM-Interleaved generations (with LLM Constraints and with Gold Constraints) on VWP narratives.