Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal interpretable automatic video captioning

Antoine Hanna-Asaad, Decky Aspandi, Titus Zaharia

TL;DR

This work introduces a novel video captioning method trained with multi-modal contrastive loss that emphasizes both multi-modal integration and interpretability, designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions.

Abstract

Video captioning aims to describe video contents using natural language format that involves understanding and interpreting scenes, actions and events that occurs simultaneously on the view. Current approaches have mainly concentrated on visual cues, often neglecting the rich information available from other important modality of audio information, including their inter-dependencies. In this work, we introduce a novel video captioning method trained with multi-modal contrastive loss that emphasizes both multi-modal integration and interpretability. Our approach is designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions. Furthermore, we highlight the importance of interpretability, employing multiple attention mechanisms that provide explanation into the model's decision-making process. Our experimental results demonstrate that our proposed method performs favorably against the state-of the-art models on commonly used benchmark datasets of MSR-VTT and VATEX.

Multi-Modal interpretable automatic video captioning

TL;DR

This work introduces a novel video captioning method trained with multi-modal contrastive loss that emphasizes both multi-modal integration and interpretability, designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions.

Abstract

Video captioning aims to describe video contents using natural language format that involves understanding and interpreting scenes, actions and events that occurs simultaneously on the view. Current approaches have mainly concentrated on visual cues, often neglecting the rich information available from other important modality of audio information, including their inter-dependencies. In this work, we introduce a novel video captioning method trained with multi-modal contrastive loss that emphasizes both multi-modal integration and interpretability. Our approach is designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions. Furthermore, we highlight the importance of interpretability, employing multiple attention mechanisms that provide explanation into the model's decision-making process. Our experimental results demonstrate that our proposed method performs favorably against the state-of the-art models on commonly used benchmark datasets of MSR-VTT and VATEX.

Paper Structure

This paper contains 17 sections, 15 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-Modal Captioning
  4. Explainability
  5. Methodology
  6. Modalities Encoders
  7. Modalities Combiner
  8. Caption Decoder
  9. Loss Functions and Models Training
  10. Implementation Details
  11. Experiment Results
  12. Datasets and Experiment Settings
  13. The impact of Fusion and Contrastive Loss
  14. Visual and Audio Interpretability
  15. Explorations of features space on Contrastive loss
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Overview of our proposed methods that involves both Video and Respective Audio Captions to generate a Video Caption.
  • Figure 2: Visualisation of different explainability aspect : Attention weights on the image, audio caption and generated video caption respectively.
  • Figure 3: Visualization of t-SNE : Fused vs FullModel (MICap)
  • Figure 4: Vizualisation of different explainability aspect, on different models : Attention weights on the image, audio caption and caption prediction respectively