VidTune: Creating Video Soundtracks with Generative Music and Contextual Thumbnails

TL;DR

VidTune tackles the difficulty of aligning video soundtracks with narrative mood by introducing an interactive system that expands prompts to generate diverse music options and grounds evaluation with contextual thumbnails anchored in the video. It combines prompt expansion, contextual thumbnail generation, and a music-space map to enable scalable sensemaking, exploration, and refinement. Technical evaluations show thumbnails better reflect musical attributes than baselines, and a controlled study demonstrates improved exploration, review efficiency, and user satisfaction with VidTune, with additional insights from an exploratory case study. Collectively, the work demonstrates how making music more visual and prompt-driven can shift soundtrack selection from a retrieval task to a creative, engaging exploration process, expanding accessibility for diverse creator populations.

Abstract

Music shapes the tone of videos, yet creators often struggle to find soundtracks that match their video's mood and narrative. Recent text-to-music models let creators generate music from text prompts, but our formative study (N=8) shows creators struggle to construct diverse prompts, quickly review and compare tracks, and understand their impact on the video. We present VidTune, a system that supports soundtrack creation by generating diverse music options from a creator's prompt and producing contextual thumbnails for rapid review. VidTune extracts representative video subjects to ground thumbnails in context, maps each track's valence and energy onto visual cues like color and brightness, and depicts prominent genres and instruments. Creators can refine tracks through natural language edits, which VidTune expands into new generations. In a controlled user study (N=12) and an exploratory case study (N=6), participants found VidTune helpful for efficiently reviewing and comparing music options and described the process as playful and enriching.

Figures (9)

• Figure 1: VidTune Interface: The video player and timeline let users choose a scene to add music and preview candidates in sync with the video (A-C). VidTune surfaces prompt suggestions based on the selected scene and user goal (D), then expands the prompt to generate 4 candidates with contextual thumbnails (E). On hover, users see reusable prompt keywords (F) and a fit check (G). Users can iterate with natural-language-edits (H), organize generations via filter/search (I), or view a music map for similarity-based exploration (J).
• Figure 2: VidTune's animated thumbnail sampled at 0.5 FPS. As the thumbnail animates, the character switches to instruments that enter later in the audio (A, B), movements convey rough tempo (C), and visual effects convey energy (D).
• Figure 3: VidTune’s Music Map arranges generated tracks in a 2D space by audio-embedding similarity (CLAP wu2023large), revealing families of related music at a glance. A dashed path shows the sequence of tracks added to the current video. Users can multi-select tracks and use Blend to generate similar variations.
• Figure 4: From a user video, VidTune extracts an anchor subject and description to form a base prompt grounded in the footage. It analyzes the generated music to infer musical attributes, maps them to visual cues, and fuses these into a style prompt. The fused prompt is used to generate static and animated thumbnails that reflect both the video context and the music.
• Figure 5: Example thumbnails from the pipeline evaluation study. For each prompt (column), the top row shows VidTune thumbnails and the bottom row shows the baseline. VidTune tends to convey music‐relevant cues (e.g., genre, instruments), while the baseline leans toward literal prompt imagery or abstract scenes.