Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contextual AD Narration with Interleaved Multimodal Sequence

Hanlin Wang, Zhan Tong, Kecheng Zheng, Yujun Shen, Limin Wang

TL;DR

This work addresses automatic Audio Description generation for long-form video by introducing Uni-AD, which prompts frozen LLMs with interleaved multimodal sequences that fuse video frames, character exemplars, text, and context. It couples a visual mapping network to align multi-modal content, a character-refinement module to isolate AD-relevant characters, and contextual information plus a contrastive loss to improve coherence and diversity. Empirical results on MAD-eval-Named, CMDAD, and TVAD show state-of-the-art performance and robustness across settings, with improvements from stronger visual encoders and larger LLMs. The approach is scalable and accompanies public code to facilitate reproducibility and broader adoption in AD generation research.

Abstract

The Audio Description (AD) task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies. With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name and provide reasonable, contextual descriptions to help audience understand the storyline of movie. To achieve this goal, we propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs with interleaved multimodal sequence as input, termed as Uni-AD. To enhance the alignment of features across various modalities with finer granularity, we introduce a simple and lightweight module that maps video features into the textual feature space. Moreover, we also propose a character-refinement module to provide more precise information by identifying the main characters who play more significant roles in the video context. With these unique designs, we further incorporate contextual information and a contrastive loss into our architecture to generate smoother and more contextually appropriate ADs. Experiments on multiple AD datasets show that Uni-AD performs well on AD generation, which demonstrates the effectiveness of our approach. Our code is available at: https://github.com/ant-research/UniAD.

Contextual AD Narration with Interleaved Multimodal Sequence

TL;DR

This work addresses automatic Audio Description generation for long-form video by introducing Uni-AD, which prompts frozen LLMs with interleaved multimodal sequences that fuse video frames, character exemplars, text, and context. It couples a visual mapping network to align multi-modal content, a character-refinement module to isolate AD-relevant characters, and contextual information plus a contrastive loss to improve coherence and diversity. Empirical results on MAD-eval-Named, CMDAD, and TVAD show state-of-the-art performance and robustness across settings, with improvements from stronger visual encoders and larger LLMs. The approach is scalable and accompanies public code to facilitate reproducibility and broader adoption in AD generation research.

Abstract

The Audio Description (AD) task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies. With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name and provide reasonable, contextual descriptions to help audience understand the storyline of movie. To achieve this goal, we propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs with interleaved multimodal sequence as input, termed as Uni-AD. To enhance the alignment of features across various modalities with finer granularity, we introduce a simple and lightweight module that maps video features into the textual feature space. Moreover, we also propose a character-refinement module to provide more precise information by identifying the main characters who play more significant roles in the video context. With these unique designs, we further incorporate contextual information and a contrastive loss into our architecture to generate smoother and more contextually appropriate ADs. Experiments on multiple AD datasets show that Uni-AD performs well on AD generation, which demonstrates the effectiveness of our approach. Our code is available at: https://github.com/ant-research/UniAD.
Paper Structure (31 sections, 5 equations, 6 figures, 14 tables)

This paper contains 31 sections, 5 equations, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio Descriptions Generation
  4. Interleaved Sequence for Multimodal Learning
  5. LLMs for Video Understanding
  6. Methodology
  7. Method Overview
  8. Uni-AD Pipeline
  9. Character-Refinement Module
  10. Contextual Information Modeling
  11. Objective Function
  12. Experiments
  13. Implementation Details
  14. Comparison with state-of-the-art approaches
  15. Ablation Study
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Taking video clip, text, character bank and context information as the inputs, the narrator generates corresponding audio description (AD) for video comprehension. Rather than describe all characters appearing in the video, the narrator should focus on characters that truly contribute to the storyline.
  • Figure 2: Overall architecture of our proposed Uni-AD. Our model first filters the input character information to retain the AD-related characters. Then all visual contents are mapped into the unified semantic space to form the interleaved multimodal sequence with text and contextual information. Afterwards, we prompt a frozen LLM with this sequence to generate the corresponding AD.
  • Figure 3: Structure of Visual Mapping Network and Character-Refinement Module.
  • Figure 4: Qualitative analysis on character-refinement module, contextual information, number of learnable vectors and comparison with other approaches. Movies are selected from (a): How Do You Know(2010), (bc): Legion(2010), (d): Charlie St. Cloud (2010).
  • Figure S1: Qualitative analysis on character-refinement module and contextual information. Movies are selected from (a): Signs (2002), (b): The Ides of March (2011).
  • ...and 1 more figures