SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities

TL;DR

A novel Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities (SPECTRUM) framework to empower the generation of emotionally and semantically credible captions.

Abstract

Capturing a video's meaning and critical concepts by analyzing the subtle details is a fundamental yet challenging task in video captioning. Identifying the dominant emotional tone in a video significantly enhances the perception of its context. Despite a strong emphasis on video captioning, existing models often need to adequately address emotional themes, resulting in suboptimal captioning results. To address these limitations, this paper proposes a novel Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities (SPECTRUM) framework to empower the generation of emotionally and semantically credible captions. Leveraging our pioneering structure, SPECTRUM discerns multimodal semantics and emotional themes using Visual Text Attribute Investigation (VTAI) and determines the orientation of descriptive captions through a Holistic Concept-Oriented Theme (HCOT), expressing emotionally-informed and field-acquainted references. They exploit video-to-text retrieval capabilities and the multifaceted nature of video content to estimate the emotional probabilities of candidate captions. Then, the dominant theme of the video is determined by appropriately weighting embedded attribute vectors and applying coarse- and fine-grained emotional concepts, which define the video's contextual alignment. Furthermore, using two loss functions, SPECTRUM is optimized to integrate emotional information and minimize prediction errors. Extensive experiments on the EmVidCap, MSVD, and MSRVTT video captioning datasets demonstrate that our model significantly surpasses state-of-the-art methods. Quantitative and qualitative evaluations highlight the model's ability to accurately capture and convey video emotions and multimodal attributes.

Figures (3)

• Figure 1: Diagram of the SPECTRUM framework illustrating the caption generation procedure of a query video based on the emotional spectrum. The SPECTRUM architecture is designed to generate emotionally-informed and factually-acquainted captions through the use of a concept investigation unit (CIU), including video-text attribute investigation (VTAI) and holistic concept-oriented theme (HCOT).
• Figure 2: Overview of the proposed SPECTRUM. Our model utilizes the retrieved textual features with video signals to generate accurate captions. Given $\mathbf{F}_{(enc)}$ as a result of concatenation of multimodal features and $\mathbf{A}_{(emb)}$ (Key matrix and Value matrix) extracted from the predicted probability of attribute concept $Pr^{(ATT)}$, the Pre-LN Transformer generates emotionally-informed and factually-acquainted captions for accurate description of video contents.
• Figure 3: Captioning examples from EmVidCap Dataset, including two human-annotated ground-truth captions, captions generated by the Baseline and proposed SPECTRUM models, and bar charts showing the attention score of factual words over detected concepts. Accurate factual concepts, and wrongly predicted concepts are pointed out.