LLM-AD: Large Language Model based Audio Description System

TL;DR

This work delivers a training-free, GPT-4V-based pipeline for automated Audio Description generation that couples visual frame analysis with a tracking-based character recognition module to maintain narrative coherence across frames. By leveraging explicit AD guidelines, framing prompts to enforce AD style and word-count, and incorporating textual context from subtitles, the approach achieves competitive performance on the MAD dataset (CIDEr $20.5$; ROUGE-L $13.5$) and outperforms prior AutoAD methods. Key contributions include the tracking-driven identity resolution without training and a thorough ablation study on visual prompts, textual context, and AD-length control. The results suggest a scalable path toward accessible, eyes-free video consumption, while highlighting biases and the need for automatic AD insertion timing and length estimation as future work.

Abstract

The development of Audio Description (AD) has been a pivotal step forward in making video content more accessible and inclusive. Traditionally, AD production has demanded a considerable amount of skilled labor, while existing automated approaches still necessitate extensive training to integrate multimodal inputs and tailor the output from a captioning style to an AD style. In this paper, we introduce an automated AD generation pipeline that harnesses the potent multimodal and instruction-following capacities of GPT-4V(ision). Notably, our methodology employs readily available components, eliminating the need for additional training. It produces ADs that not only comply with established natural language AD production standards but also maintain contextually consistent character information across frames, courtesy of a tracking-based character recognition module. A thorough analysis on the MAD dataset reveals that our approach achieves a performance on par with learning-based methods in automated AD production, as substantiated by a CIDEr score of 20.5.

Figures (6)

• Figure 1: Overview of the GPT-4V based automated AD generation pipeline.
• Figure 2: Use GPT-4V for character recognition.
• Figure 3: Illustration of different visual prompting to add character information.
• Figure 4: Generated AD with different character recognition methods.
• Figure 5: Generated AD with different linguistic styles instructions.