Extending Visual Dynamics for Video-to-Music Generation

TL;DR

This work tackles video-to-music generation by addressing two core challenges: capturing fine-grained visual dynamics and achieving precise temporal alignment between video frames and music tokens. It introduces DyViM, a framework that extracts frame-wise dynamics with a motion-encoder inspired by optical-flow methods and uses self-attention to produce dense dynamic features, which are then used to extend music tokens for token-level synchronization. High-level semantics are provided via cross-attention to a pre-trained image encoder, ensuring thematic coherence, while an annealing tuning strategy enables efficient adaptation of a pre-trained music decoder. Extensive experiments on three datasets show DyViM achieving state-of-the-art performance across both objective and subjective metrics, validating the importance of dynamics-aware conditioning in video-to-music generation.

Abstract

Music profoundly enhances video production by improving quality, engagement, and emotional resonance, sparking growing interest in video-to-music generation. Despite recent advances, existing approaches remain limited in specific scenarios or undervalue the visual dynamics. To address these limitations, we focus on tackling the complexity of dynamics and resolving temporal misalignment between video and music representations. To this end, we propose DyViM, a novel framework to enhance dynamics modeling for video-to-music generation. Specifically, we extract frame-wise dynamics features via a simplified motion encoder inherited from optical flow methods, followed by a self-attention module for aggregation within frames. These dynamic features are then incorporated to extend existing music tokens for temporal alignment. Additionally, high-level semantics are conveyed through a cross-attention mechanism, and an annealing tuning strategy benefits to fine-tune well-trained music decoders efficiently, therefore facilitating seamless adaptation. Extensive experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.

Figures (47)

• Figure 1: Illustrative examples of video-music pairs of dance video (left) and music video (right). Visual dynamics (e.g., body motions or scene transitions) demonstrates temporal synchronization with music rhythm and changes (e.g., beats).
• Figure 2: The overall framework of DyViM. We enhance the dynamics modeling by extracting frame-wise dynamics features and interpolating them to extend music tokens to achieve token-level conditioning. Additionally, keyframes provide semantics with cross-attention. An annealing tuning strategy is employed to optimize the model in an effective and efficient manner.
• Figure 3: Performance comparison (bar) and the time cost during training (line) w.r.t. different tuning strategies.
• Figure 4: Performance comparison w.r.t. different $\alpha$ to control the strength of visual dynamics.
• Figure 5: An illustration of the dynamics attention matrix for dance videos (top) and music videos (bottom), where the size of each point represents the quantitative value of attention.