Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition

TL;DR

A novel method that measures story quality in terms of human likeness regarding three key aspects of visual grounding, coherence, and repetitiveness is introduced, whose results suggest that a 'good' story may require more than a human-like level of visual grounding, coherence, and repetition.

Abstract

Visual storytelling consists in generating a natural language story given a temporally ordered sequence of images. This task is not only challenging for models, but also very difficult to evaluate with automatic metrics since there is no consensus about what makes a story 'good'. In this paper, we introduce a novel method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. We then use this method to evaluate the stories generated by several models, showing that the foundation model LLaVA obtains the best result, but only slightly so compared to TAPM, a 50-times smaller visual storytelling model. Upgrading the visual and language components of TAPM results in a model that yields competitive performance with a relatively low number of parameters. Finally, we carry out a human evaluation study, whose results suggest that a 'good' story may require more than a human-like level of visual grounding, coherence, and repetition.

Figures (12)

• Figure 1: Distance between human- and model-generated stories in the VIST test set according to our proposed measure $\text{d}_{HM}$ (the lower the better). For BLIP-2 and LLaVA, the best setting is reported; results for all settings are provided in Fig. \ref{['fig:1sup']}, Appendix \ref{['appendix:s4']}.
• Figure 2: An example from the VIST test set with corresponding model-generated stories and scores from individual evaluation metrics (Coherence: C, Visual grounding: G, Repetition: R).
• Figure 3: For the TAPM model and its upgraded versions (+Llama 2) and (+ViT), we compute the $\text{d}_{HM}$ values on two datasets---VIST and VWP.
• Figure 4: Comparison between the number of model parameters and their corresponding $\text{d}_{HM}$ values on the VIST test set (the lower the better). We observe that as the models increase in size, the scores for the stories they generated get closer to scores of human annotations.
• Figure 5: Aggregated judgments (5 human evaluators per model) comparing human stories in the VIST test set to stories generated by the two best-performing models according to our distance measure $\text{d}_{HM}$.