Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality

Sishuo Chen, Lei Li, Shuhuai Ren, Rundong Gao, Yuanxin Liu, Xiaohan Bi, Xu Sun, Lu Hou

TL;DR

A Missing-Resistant framework MR-VPC is proposed that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities, and has proven to deliver superior performance on modality-complete and modality-missing test data.

Abstract

Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries. However, the existing models are constrained by the assumption of constant availability of a single auxiliary modality, which is impractical given the diversity and unpredictable nature of real-world scenarios. To this end, we propose a Missing-Resistant framework MR-VPC that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities. Under this framework, we propose the Multimodal VPC (MVPC) architecture integrating video, speech, and event boundary inputs in a unified manner to process various auxiliary inputs. Moreover, to fortify the model against incomplete data, we introduce DropAM, a data augmentation strategy that randomly omits auxiliary inputs, paired with DistillAM, a regularization target that distills knowledge from teacher models trained on modality-complete data, enabling efficient learning in modality-deficient environments. Through exhaustive experimentation on YouCook2 and ActivityNet Captions, MR-VPC has proven to deliver superior performance on modality-complete and modality-missing test data. This work highlights the significance of developing resilient VPC models and paves the way for more adaptive, robust multimodal video understanding.

Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality

TL;DR

A Missing-Resistant framework MR-VPC is proposed that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities, and has proven to deliver superior performance on modality-complete and modality-missing test data.

Abstract

Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries. However, the existing models are constrained by the assumption of constant availability of a single auxiliary modality, which is impractical given the diversity and unpredictable nature of real-world scenarios. To this end, we propose a Missing-Resistant framework MR-VPC that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities. Under this framework, we propose the Multimodal VPC (MVPC) architecture integrating video, speech, and event boundary inputs in a unified manner to process various auxiliary inputs. Moreover, to fortify the model against incomplete data, we introduce DropAM, a data augmentation strategy that randomly omits auxiliary inputs, paired with DistillAM, a regularization target that distills knowledge from teacher models trained on modality-complete data, enabling efficient learning in modality-deficient environments. Through exhaustive experimentation on YouCook2 and ActivityNet Captions, MR-VPC has proven to deliver superior performance on modality-complete and modality-missing test data. This work highlights the significance of developing resilient VPC models and paves the way for more adaptive, robust multimodal video understanding.
Paper Structure (50 sections, 2 equations, 6 figures, 14 tables)

This paper contains 50 sections, 2 equations, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Paragraph Captioning (VPC)
  4. Robustness to Missing Modality
  5. Methodology
  6. Problem Formulation
  7. MVPC Model Framework
  8. Overview
  9. Video Encoder
  10. Text Encoder
  11. Fusion Module and Text Decoder
  12. Weight Initialization
  13. Training Strategies of MR-VPC
  14. DropAM: Drop Auxiliary Modalities
  15. DistillAM: Learning from the Teacher with Modality-Complete Data
  16. ...and 35 more sections

Figures (6)

  • Figure 1: The performance of the previous SOTA model Vid2Seq drastically declines as the percentage of ASR text missing grows. In contrast, our MR-VPC consistently achieves superior performance in both modality-complete and modality-missing environments.
  • Figure 2: The composition of an instance in the multimodal VPC task from the validation set of YouCook2.
  • Figure 3: The overview diagram of our MVPC (multimodal video paragraph captioning) framework.
  • Figure 4: Visualization of the SimCSE embeddings of the captions generated under modality-complete and modality-missing (video-only) scenarios.
  • Figure 5: The captions produced by our models and baselines in the modality-complete setting on an ActivityNet Captions test sample (id: "bXdq2zI1Ms0"). The wrongly predicted concepts are highlighted in red by the author.
  • ...and 1 more figures