Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Abstractive Summarization for How2 Videos

Shruti Palaskar, Jindrich Libovický, Spandana Gella, Florian Metze

TL;DR

This work tackles open-domain video summarization by producing fluent textual summaries that fuse video content and speech transcripts. It proposes a multimodal abstractive framework with hierarchical attention to integrate text and video features and introduces Content F1 as a semantic, content-focused evaluation metric. Experiments on the How2 dataset show that multimodal (text+video) models outperform unimodal baselines, with transfer learning enabling cross-dataset gains. The results highlight the value of content-centric evaluation for teaser-style video summaries and point to future directions for end-to-end audio-based and multi-video summarization.

Abstract

In this paper, we study abstractive summarization for open-domain videos. Unlike the traditional text news summarization, the goal is less to "compress" text information but rather to provide a fluent textual summary of information that has been collected and fused from different source modalities, in our case video and audio transcripts (or text). We show how a multi-source sequence-to-sequence model with hierarchical attention can integrate information from different modalities into a coherent output, compare various models trained with different modalities and present pilot experiments on the How2 corpus of instructional videos. We also propose a new evaluation metric (Content F1) for abstractive summarization task that measures semantic adequacy rather than fluency of the summaries, which is covered by metrics like ROUGE and BLEU.

Multimodal Abstractive Summarization for How2 Videos

TL;DR

This work tackles open-domain video summarization by producing fluent textual summaries that fuse video content and speech transcripts. It proposes a multimodal abstractive framework with hierarchical attention to integrate text and video features and introduces Content F1 as a semantic, content-focused evaluation metric. Experiments on the How2 dataset show that multimodal (text+video) models outperform unimodal baselines, with transfer learning enabling cross-dataset gains. The results highlight the value of content-centric evaluation for teaser-style video summaries and point to future directions for end-to-end audio-based and multi-video summarization.

Abstract

In this paper, we study abstractive summarization for open-domain videos. Unlike the traditional text news summarization, the goal is less to "compress" text information but rather to provide a fluent textual summary of information that has been collected and fused from different source modalities, in our case video and audio transcripts (or text). We show how a multi-source sequence-to-sequence model with hierarchical attention can integrate information from different modalities into a coherent output, compare various models trained with different modalities and present pilot experiments on the How2 corpus of instructional videos. We also propose a new evaluation metric (Content F1) for abstractive summarization task that measures semantic adequacy rather than fluency of the summaries, which is covered by metrics like ROUGE and BLEU.

Paper Structure

This paper contains 16 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Multimodal Abstractive Summarization
  3. Video-based Summarization.
  4. Speech-based Summarization.
  5. Transfer Learning.
  6. Evaluation
  7. Content F1.
  8. Human Evaluation.
  9. Experiments and Results
  10. Conclusions
  11. Appendix
  12. Experimental Setup
  13. Frequent Words in Transcripts and Summaries
  14. Output Examples from Different Models
  15. Attention Analysis
  16. ...and 1 more sections

Figures (4)

  • Figure 1: How2 dataset example with different modalities. "Cuban breakfast" and "free cooking video" is not mentioned in the transcript, and has to be derived from other sources.
  • Figure 2: Building blocks of the sequence-to-sequence models, gray numbers in brackets indicate which components are utilized in which experiments.
  • Figure 3: Word distribution in comparison with the human summaries for different unimodal and multimodal models. Density curves show the length distributions of human annotated and system produced summaries.
  • Figure A1: Visualizing Attention over Video Features.