Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model

Yong Ren, Chenxing Li, Le Xu, Hao Gu, Duzhen Zhang, Yujie Chen, Manjie Xu, Ruibo Fu, Shan Yang, Dong Yu

TL;DR

The work defines SVAD to probe vision-language models' ability to reason about non-visible audio from silent videos, addressing a key gap in VT2A workflows. It introduces a CoT-based supervised fine-tuning paradigm and the CoT-AudioCaps dataset to structure reasoning from visual inputs to audio captions, leveraging LoRA adapters for efficient adaptation. Experiments show that pre-trained VLMs struggle on SVAD, while single-stage SFT and especially CoT-SFT substantially improve modal-mismatch reasoning and reduce audio-description acquisition challenges during VT2A inference. The findings offer a path toward more human-like multimodal reasoning in the absence of target modalities, with practical implications for video Foley and related audio-caption alignment tasks.

Abstract

Humans can intuitively infer sounds from silent videos, but whether multimodal large language models can perform modal-mismatch reasoning without accessing target modalities remains relatively unexplored. Current text-assisted-video-to-audio (VT2A) methods excel in video foley tasks but struggle to acquire audio descriptions during inference. We introduce the task of Reasoning Audio Descriptions from Silent Videos (SVAD) to address this challenge and investigate vision-language models' (VLMs) capabilities on this task. To further enhance the VLMs' reasoning capacity for the SVAD task, we construct a CoT-AudioCaps dataset and propose a Chain-of-Thought-based supervised fine-tuning strategy. Experiments on SVAD and subsequent VT2A tasks demonstrate our method's effectiveness in two key aspects: significantly improving VLMs' modal-mismatch reasoning for SVAD and effectively addressing the challenge of acquiring audio descriptions during VT2A inference.

Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model

TL;DR

The work defines SVAD to probe vision-language models' ability to reason about non-visible audio from silent videos, addressing a key gap in VT2A workflows. It introduces a CoT-based supervised fine-tuning paradigm and the CoT-AudioCaps dataset to structure reasoning from visual inputs to audio captions, leveraging LoRA adapters for efficient adaptation. Experiments show that pre-trained VLMs struggle on SVAD, while single-stage SFT and especially CoT-SFT substantially improve modal-mismatch reasoning and reduce audio-description acquisition challenges during VT2A inference. The findings offer a path toward more human-like multimodal reasoning in the absence of target modalities, with practical implications for video Foley and related audio-caption alignment tasks.

Abstract

Humans can intuitively infer sounds from silent videos, but whether multimodal large language models can perform modal-mismatch reasoning without accessing target modalities remains relatively unexplored. Current text-assisted-video-to-audio (VT2A) methods excel in video foley tasks but struggle to acquire audio descriptions during inference. We introduce the task of Reasoning Audio Descriptions from Silent Videos (SVAD) to address this challenge and investigate vision-language models' (VLMs) capabilities on this task. To further enhance the VLMs' reasoning capacity for the SVAD task, we construct a CoT-AudioCaps dataset and propose a Chain-of-Thought-based supervised fine-tuning strategy. Experiments on SVAD and subsequent VT2A tasks demonstrate our method's effectiveness in two key aspects: significantly improving VLMs' modal-mismatch reasoning for SVAD and effectively addressing the challenge of acquiring audio descriptions during VT2A inference.
Paper Structure (14 sections, 4 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 14 sections, 4 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. SFT for SVAD
  4. CoT-AudioCaps: Dataset Construction
  5. CoT-based SFT: SFT strategy
  6. Experiments
  7. Experimental Settings
  8. Datasets and Baselines
  9. Metrics
  10. Implementation Details.
  11. Eval Pre-training VLMs in SVAD (RQ1)
  12. SFT and CoT-based SFT (RQ2)
  13. VT2A inference (RQ3)
  14. Conclusion

Figures (4)

  • Figure 1: Good sound descriptions from humans. vs. auditory-irrelevant hallucination from VLMs when reason audio descriptions from silent videos.
  • Figure 2: Two primary technical paradigms of video foley and challenges faced by VT2A.
  • Figure 3: Overview of our methods for SVAD task, including two SFT strategies, the SFT training for VLM by LoRA, the CoT-Audiocaps Dataset construction process, and the CoT-based SFT method for SVAD.
  • Figure 4: The templates for constructing the CoT-Audiocaps Dataset, the direct prompt template for video and audio caption from video for VLMs, and the CoT prompt template for VLMs (For LLMs, replace the video with the video caption).