Soundify: Matching Sound Effects to Video

TL;DR

Soundify addresses the time-consuming Foley workflow by retrieving high-quality sound clips from studio libraries and localizing them to video using CLIP-based open-vocabulary detection and Grad-CAM activation maps. It surfaces candidate sounds, synchronizes them to on-screen emitters, and dynamically adjusts pan and gain to produce spatial audio, while enabling multi-track stacking. Evaluation across 889 human raters and 12 professional editors shows Soundify outperforms a YOLO-based baseline in perceived audio-video alignment, reduces editor workload and task time, and improves usability. The approach demonstrates a practical, library-driven alternative to generative audio synthesis for video editing, with strong potential for integration into professional workflows.

Abstract

In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Figures (10)

• Figure 1: Overview of Soundify. Soundify first splits a video into scenes. For each scene, Soundify classifies for effects and ambients. The matched ambient is used for the entire scene. For each matched effect, Soundify performs more fine-grained synchronization by identifying their appearing intervals. For each interval, Soundify mixes spatial sound chunks with computed pan and gain parameters. The final result consists of one or more effects tracks and an ambients track.
• Figure 2: Effects classification. Given the frames of a scene and a database of sound labels, Soundify performs pairwise comparisons to predict the top-5 matching sounds.
• Figure 3: Ambients classification. Since ambients classification can be more error-prone, given the user-select effects label and predicted ambients labels, Soundify performs pairwise comparisons to rerank the ambients.
• Figure 4: Sync. Given the frames of a scene and a sound label, Soundify identifies appearing intervals. An interval is split into chunks. Each chunk takes the first frame as its reference frame.
• Figure 5: Soundify adapts pan (top row) and gain (bottom row) parameters over time based on the heatmap's position and size.