NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative

TL;DR

This work tackles the absence of causal-temporal narrative in video captions by introducing NarrativeBridge, a framework that comprises a Causal-Temporal Narrative (CTN) captions benchmark and a dedicated two-stage Cause-Effect Network (CEN). The CTN benchmark is generated via large-language-model prompts with automatic EMScore filtering and human validation, producing captions that encode cause-effect relations and event sequences. The CEN architecture separately encodes cause and effect dynamics in Stage 1 and then fuses these representations in Stage 2 to generate CTN captions, trained with a combination of contrastive and captioning losses. Experiments on MSVD-CTN and MSRVTT-CTN demonstrate that CEN outperforms state-of-the-art captioning models and vision-language models in articulating causal-temporal narratives, with strong cross-dataset generalization and thorough ablations validating the design. The work also provides a pathway for labeling unlabeled videos and advancing practically relevant video understanding tasks through enriched, narrative-capable captions.

Abstract

Existing video captioning benchmarks and models lack causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models' ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, we propose NarrativeBridge, an approach comprising of: (1) a novel Causal-Temporal Narrative (CTN) captions benchmark generated using a large language model and few-shot prompting, explicitly encoding cause-effect temporal relationships in video descriptions; and (2) a Cause-Effect Network (CEN) with separate encoders for capturing cause and effect dynamics, enabling effective learning and generation of captions with causal-temporal narrative. Extensive experiments demonstrate that CEN significantly outperforms state-of-the-art models in articulating the causal and temporal aspects of video content: 17.88 and 17.44 CIDEr on the MSVD-CTN and MSRVTT-CTN datasets, respectively. Cross-dataset evaluations further showcase CEN's strong generalization capabilities. The proposed framework understands and generates nuanced text descriptions with intricate causal-temporal narrative structures present in videos, addressing a critical limitation in video captioning. For project details, visit https://narrativebridge.github.io/.

Figures (31)

• Figure 1: Comparison of Original captions vs. Causal-Temporal Narrative (CTN) caption to illustrate the inclusion of causal-temporal narrative.
• Figure 2: CTN caption generation pipeline. $\theta$ indicates a threshold.
• Figure 3: The two-stage Cause-Effect Network (CEN) architecture. Stage 1: Separate Cause ($E_{cause}$) and Effect ($E_{effect}$) video encoders, pretrained using CLIP-ViT, learn specialized video representations. Corresponding text encoders ($T_{cause}$ and $T_{effect}$) encode the cause and effect portions of the CTN caption. Contrastive losses are applied to align the video and text embeddings. Stage 2: The learned cause and effect video features are encoded separately ($Enc_{cause}$ and $Enc_{effect}$) and concatenated before being input to the decoder, which generates the final CTN caption.
• Figure 4: (a) UMAP visualization of video features learned from CTN (red) and original (blue) captions on MSR-VTT, showing non-overlapping feature spaces. (b) UMAP visualization of video features learned from cause (black) and effect (orange) parts of CTN captions on MSRVTT-CTN, showing near-complete overlap.
• Figure 5: Ablation study results on the MSVD-CTN and MSRVTT-CTN datasets. $\text{E}_\text{combined}$, w/o FT CLIPs, Only $\text{E}_\text{cause}$ and Only $\text{E}_\text{effect}$ are baselines of our CEN architecture, while Zero Shot X and Fine-tune X represent cross-dataset evaluation settings. The best results in each category are in bold.