Glow with the Flow: AI-Assisted Creation of Ambient Lightscapes for Music Videos

TL;DR

The paper addresses barriers to designed lighting in consumer music-video contexts by introducing an AI-assisted workflow that generates ambient lightscapes. It combines multimodal analysis of audio and video with rule-based synthesis to produce editable, object-based light objects aligned with professional design heuristics. An evaluation with three music videos and 32 participants shows AI-generated first drafts achieve perceptual parity with hand-authored designs across emotional congruence, rhythm, and color, supporting their use as viable baselines for refinement. The work demonstrates the potential of AI-assisted, co-creative workflows to broaden adoption of immersive lighting beyond professional venues.

Abstract

Designed light is an established modality for live performance and music playback. Despite the growing availability of consumer smart lighting, the creation of designed light for music visualization remains limited to professional contexts due to time and skill constraints. To address this, we present an AI-assisted system for generating ambient light sequences for music videos. Informed by professional design heuristics, the system extracts salient features from source video and audio to generate an editable preliminary design of object based ambient light effect. We evaluated the system by comparing its autonomous output against hand-authored designs for three music videos. Findings from responses by 32 participants indicate that the initial output provides a viable baseline for further refinement by human authors. This work demonstrates the utility of AI-assisted workflows in supporting the creation and adoption of designed light beyond professional venues.

Figures (5)

• Figure 1: Mapping out the lightscape design process whereby a creative uses multimodal information from a music video to inform the creation of an accompanying light design. Video information determines an initial color palette which is then enriched to meet various functional requirements. Audio information on an event-, section-, and song-level then determines when light events occur.
• Figure 2: Conceptual flow of proposed system. On an input music video, we perform a multimodal AI which create cues for light objects. This light object representation in editable in authoring tool and rendered into the available light fixtures using southwell2025creating.
• Figure 3: Light content visualization for a music video excerpt.
• Figure 4: Box plots of individual participant ratings for AI-generated and human-generated outputs across 15 contexts and 32 participants for each perceptual attribute (emotional congruence, rhythmic synchronization, and chromatic congruence). Each box summarizes the distribution of ratings across all individual observations, illustrating the substantial overlap between AI and human conditions for all three attributes.
• Figure 5: Visualizing the light designs of three songs, with each showing hand-authored and generated versions. The x axis shows frames along the songs' timeline, and the y axis shows the ratios of colors present during a given frame. Note how generated versions generally reflect section-structure, while having reduced color palette.