Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling the Invisible: Captioning Videos with Metaphors

Abisek Rajakumar Kalarani, Pushpak Bhattacharyya, Sumit Shekhar

TL;DR

This work introduces a new VL task of describing the metaphors present in the videos and proposes a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task.

Abstract

Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent studies have shown that Vision-Language (VL) models cannot understand visual metaphors in memes and adverts. As of now, no probing studies have been done that involve complex language phenomena like metaphors with videos. Hence, we introduce a new VL task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a manually created dataset with 705 videos and 2115 human-written captions, along with a new metric called Average Concept Distance (ACD), to automatically evaluate the creativity of the metaphors generated. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task. We perform a comprehensive analysis of existing video language models on this task and publish our dataset, models, and benchmark results to enable further research.

Unveiling the Invisible: Captioning Videos with Metaphors

TL;DR

This work introduces a new VL task of describing the metaphors present in the videos and proposes a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task.

Abstract

Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent studies have shown that Vision-Language (VL) models cannot understand visual metaphors in memes and adverts. As of now, no probing studies have been done that involve complex language phenomena like metaphors with videos. Hence, we introduce a new VL task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a manually created dataset with 705 videos and 2115 human-written captions, along with a new metric called Average Concept Distance (ACD), to automatically evaluate the creativity of the metaphors generated. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task. We perform a comprehensive analysis of existing video language models on this task and publish our dataset, models, and benchmark results to enable further research.
Paper Structure (29 sections, 5 equations, 8 figures, 3 tables)

This paper contains 29 sections, 5 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Background
  4. Related Work
  5. Dataset
  6. Annotation Details
  7. Dataset Statistics
  8. Pretraining Dataset
  9. Our Model
  10. Experiments
  11. Pretraining
  12. Video Metaphor Captioning
  13. Evaluation Metrics
  14. Results and Analysis
  15. Automatic Metrics
  16. ...and 14 more sections

Figures (8)

  • Figure 1: An example of a creative advertisement that shows the speed of the broadband by depicting a scene from the iconic movie 'Titanic'.
  • Figure 2: Examples of metaphors used in videos to convey ideas creatively along with their explanation
  • Figure 3: An overview of our Video Metaphor Captioning system, GIT-LLaVA. In the pretraining stage, only the mapping network is trained with image datasets. In the finetuning stage, the text decoder representation of the GIT model is mapped to the embedding space of the Vicuna model to generate metaphor captions.
  • Figure 4: An overview of the "Average Concept Distance" metric. We compute the cosine distance between the primary and the secondary concept and ground it with the BERTScore.
  • Figure 5: Results of human evaluation of the captions generated by our models.
  • ...and 3 more figures