Video Echoed in Music: Semantic, Temporal, and Rhythmic Alignment for Video-to-Music Generation

TL;DR

VeM tackles video-to-music generation by marrying latent diffusion with hierarchical video parsing, enabling semantic, temporal, and rhythmic alignment. Key innovations include storyboard-guided cross-attention for multi-level semantic conditioning and a transition-beat aligner with an adaptive adapter to synchronize scene transitions with music beats. A new TB-Match dataset and novel evaluation metrics support rigorous assessment of semantic relevance and rhythmic precision, with VeM outperforming state-of-the-art baselines on both objective and subjective measures. The work advances practical, high-fidelity video-to-music generation with improved control over timing and rhythm.

Abstract

Video-to-Music generation seeks to generate musically appropriate background music that enhances audiovisual immersion for videos. However, current approaches suffer from two critical limitations: 1) incomplete representation of video details, leading to weak alignment, and 2) inadequate temporal and rhythmic correspondence, particularly in achieving precise beat synchronization. To address the challenges, we propose Video Echoed in Music (VeM), a latent music diffusion that generates high-quality soundtracks with semantic, temporal, and rhythmic alignment for input videos. To capture video details comprehensively, VeM employs a hierarchical video parsing that acts as a music conductor, orchestrating multi-level information across modalities. Modality-specific encoders, coupled with a storyboard-guided cross-attention mechanism (SG-CAtt), integrate semantic cues while maintaining temporal coherence through position and duration encoding. For rhythmic precision, the frame-level transition-beat aligner and adapter (TB-As) dynamically synchronize visual scene transitions with music beats. We further contribute a novel video-music paired dataset sourced from e-commerce advertisements and video-sharing platforms, which imposes stricter transition-beat synchronization requirements. Meanwhile, we introduce novel metrics tailored to the task. Experimental results demonstrate superiority, particularly in semantic relevance and rhythmic precision.

Figures (7)

• Figure 1: Task overview. The proposed latent music diffusion, VeM, achieves semantic, temporal, and rhythmic alignment during video-to-music generation by integrating multimodal details derived from videos as conditions.
• Figure 2: Illustration of the proposed method. The hierarchical video parsing provides a comprehensive analysis across three levels. Cross-modal features are captured by modality-specific encoders, facilitating the semantic and temporal alignment by integrating global and storyboard details into the generative latent via storyboard-guided cross-attention. The frame-level transition-beat aligner and adapter ensure precise rhythmic synchronization by coupling video scene transitions with detected music beats and adapting to the music latent.
• Figure 3: Visualized comparison shows Mel-spectrograms alongside the video frames from different methods.
• Figure 4: Implementation of hierarchical video parsing.
• Figure 5: Screenshot of manual annotation tool.