DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos

TL;DR

This work tackles the misalignment challenge in video-to-audio generation by leveraging internal chain-of-thought reasoning within a multimodal large language model to guide step-by-step audio synthesis. The DeepSound framework combines three specialized modules—audio generation, multimodal reasoning, and audio editing—operating under a four-step CoT process to detect and remove voice-over artifacts, with a silence-detection fallback for robustness. A dedicated multimodal CoT reasoning dataset supports the learning of initial thinking in audio generation, and experiments on VGGSound and a custom 18k CoT-V2A dataset demonstrate significant reductions in voice-over and improvements in distribution, quality, and temporal metrics compared to state-of-the-art baselines. The approach offers a scalable, annotation-free pathway to improve video-audio alignment with practical impact for high-fidelity, synchronized multimedia content.

Abstract

Currently, high-quality, synchronized audio is synthesized from video and optional text inputs using various multi-modal joint learning frameworks. However, the precise alignment between the visual and generated audio domains remains far from satisfactory. One key factor is the lack of sufficient temporal and semantic alignment annotations in open-source video-audio and text-audio benchmarks. Therefore, we propose a framework for audio generation from videos, leveraging the internal chain-of-thought (CoT) of a multi-modal large language model (MLLM) to enable step-by-step reasoning without requiring additional annotations. Additionally, a corresponding multi-modal reasoning dataset is constructed to facilitate the learning of initial reasoning in audio generation. In the experiments, we demonstrate the effectiveness of the proposed framework in reducing misalignment (voice-over) in generated audio and achieving competitive performance compared to various state-of-the-art models. The evaluation results show that the proposed method outperforms state-of-the-art approaches across multiple metrics. Specifically, the F DP aSST indicator is reduced by up to 10.07%, the F DP AN N s indicator by up to 11.62%, and the F DV GG indicator by up to 38.61%. Furthermore, the IS indicator improves by up to 4.95%, the IB-score indicator increases by up to 6.39%, and the DeSync indicator is reduced by up to 0.89%.

Figures (6)

• Figure 1: Current V2A models zhang2024foleycraftercheng2024tamingchen2024yingsound (upper) represent existing approaches. The proposed DeepSound (below) are designed to follow a step-by-step reasoning process to eliminate voice-over.
• Figure 2: Overview of DeepSound. The model employs a step-by-step reasoning process to generate audio from video. In the first step, it generates a coarse audio from the input video. The second step identifies voice-over components by analyzing both the coarse audio and the video. The third step removes the detected voice-over elements from the audio. Finally, the model determines whether the resulting audio is silent or not.
• Figure 3: Overview of Dual Multi-Modal Reasoning Learning.$CoT_{structure}$ represents the internal reasoning steps within the overall audio generation process. $CoT_{detail}$ refers to the step-by-step procedure for identifying voice-over components from the coarse audio and video.
• Figure 4: The voice-over labels are divided into four categories based on the presence or absence of people and human voices. The label "Yes" indicates that the sample contains voice-over, and the labels "No1", "No2", and "No3" indicate that the sample does not contain voice-over. Specifically, "No1" means the video contains neither people nor human voices, "No2" means the video contains both people and human voices, and "No3" means the video contains people but without human voices.
• Figure 5: The process flow for generating the multiple CoT dataset involves utilizing multiple models and incorporating manual verification to ensure data quality.