Semi-Supervised Image-Based Narrative Extraction: A Case Study with Historical Photographic Records

TL;DR

This paper tackles extracting coherent narratives from large historical photo collections by bridging text-based narrative methods and image data. It proposes a semi-supervised adaptation of the Narrative Maps algorithm that uses DETR-derived visual features and partial expert labels to build coherence-maximizing narratives on the ROGER Sacambaya subset (1928). The authors evaluate against expert-curated timelines of varying lengths using DTW distance and average cosine similarity, supplemented by an expert qualitative assessment, finding statistically significant improvements for longer timelines ($p<0.05$) and generally coherent storytelling. The work demonstrates the value of combining deep visual representations with graph-based narrative extraction for digital humanities, while acknowledging scalability, bias, and multimodal integration considerations for future research.

Abstract

This paper presents a semi-supervised approach to extracting narratives from historical photographic records using an adaptation of the narrative maps algorithm. We extend the original unsupervised text-based method to work with image data, leveraging deep learning techniques for visual feature extraction and similarity computation. Our method is applied to the ROGER dataset, a collection of photographs from the 1928 Sacambaya Expedition in Bolivia captured by Robert Gerstmann. We compare our algorithmically extracted visual narratives with expert-curated timelines of varying lengths (5 to 30 images) to evaluate the effectiveness of our approach. In particular, we use the Dynamic Time Warping (DTW) algorithm to match the extracted narratives with the expert-curated baseline. In addition, we asked an expert on the topic to qualitatively evaluate a representative example of the resulting narratives. Our findings show that the narrative maps approach generally outperforms random sampling for longer timelines (10+ images, p < 0.05), with expert evaluation confirming the historical accuracy and coherence of the extracted narratives. This research contributes to the field of computational analysis of visual cultural heritage, offering new tools for historians, archivists, and digital humanities scholars to explore and understand large-scale image collections. The method's ability to generate meaningful narratives from visual data opens up new possibilities for the study and interpretation of historical events through photographic evidence.

Figures (3)

• Figure 1: The proposed visual narrative extraction pipeline. We construct a coherence graph based on the content semantic similarity and partial label information of a collection of images. During extraction, users can select source and target images to extract concept narratives using the adapted narrative maps algorithm.
• Figure 2: Expert-curated narrative used as the baseline for evaluation, with images arranged in English reading order (left to right, top to bottom).
• Figure 3: Example extracted narrative using the unsupervised narrative maps algorithm, with images arranged in English reading order (left to right, top to bottom).