Combining Genre Classification and Harmonic-Percussive Features with Diffusion Models for Music-Video Generation

TL;DR

This work addresses the challenge of generating synchronized music visuals by pairing a music excerpt with user-selected artwork through diffusion models. It introduces a two-stage pipeline: first, music captioning plus genre-conditioned style descriptions guide image synthesis; second, harmonic and percussive audio energy vectors drive frame interpolation to build a music video. A novel Audio-Visual Synchrony ($AVS$) metric is introduced to quantify alignment between audio dynamics and visual transitions, evaluated against linear interpolation using $DTW$-based alignment. Results indicate higher $AVS$ values across multiple genres, suggesting improved synchronization and expressive coherence. The approach enables personalized music videos for independent artists, live performances, and public-space installations.

Abstract

This study presents a novel method for generating music visualisers using diffusion models, combining audio input with user-selected artwork. The process involves two main stages: image generation and video creation. First, music captioning and genre classification are performed, followed by the retrieval of artistic style descriptions. A diffusion model then generates images based on the user's input image and the derived artistic style descriptions. The video generation stage utilises the same diffusion model to interpolate frames, controlled by audio energy vectors derived from key musical features of harmonics and percussives. The method demonstrates promising results across various genres, and a new metric, Audio-Visual Synchrony (AVS), is introduced to quantitatively evaluate the synchronisation between visual and audio elements. Comparative analysis shows significantly higher AVS values for videos generated using the proposed method with audio energy vectors, compared to linear interpolation. This approach has potential applications in diverse fields, including independent music video creation, film production, live music events, and enhancing audio-visual experiences in public spaces.

Figures (6)

• Figure 1: The overall block diagram of the proposed approach, with two main stages of image generation and video synthesis.
• Figure 2: The audio energy vector used to control the synthesis of a 10-second music clip. (a) percussive and (b) harmonic audio time series, (c) normalised Mel-scaled spectrogram and (d) the final audio energy vector.
• Figure 3: Transitional frames generated from an initial into a target image.
• Figure 4: Generated results for each music genre with a related initial artwork image. From left to right are the music genre, initial image and generated images for each 10 seconds.
• Figure 5: Generated results using the same initial image (top) for different genres.