FoleyBench: A Benchmark For Video-to-Audio Models

TL;DR

FoleyBench addresses the gap in Foley-style V2A evaluation by introducing a 5,000-instance dataset of visually grounded, non-speech, non-music audio with matching captions, plus long-form 30-second clips. The authors implement an automated filtering and labeling pipeline to ensure audio-visual grounding, and evaluate a wide range of V2A models using metrics for audio quality and cross-modal alignment, including new insights into discrete versus ambient sounds and long-form generation. Key findings show that existing benchmarks like VGGSound are poorly aligned with Foley tasks, while FoleyBench reveals different strengths across models and highlights the critical role of text conditioning. Overall, FoleyBench provides a practical, diverse, and interpretable benchmark to drive progress in Foley-style V2A systems.

Abstract

Video-to-audio generation (V2A) is of increasing importance in domains such as film post-production, AR/VR, and sound design, particularly for the creation of Foley sound effects synchronized with on-screen actions. Foley requires generating audio that is both semantically aligned with visible events and temporally aligned with their timing. Yet, there is a mismatch between evaluation and downstream applications due to the absence of a benchmark tailored to Foley-style scenarios. We find that 74% of videos from past evaluation datasets have poor audio-visual correspondence. Moreover, they are dominated by speech and music, domains that lie outside the use case for Foley. To address this gap, we introduce FoleyBench, the first large-scale benchmark explicitly designed for Foley-style V2A evaluation. FoleyBench contains 5,000 (video, ground-truth audio, text caption) triplets, each featuring visible sound sources with audio causally tied to on-screen events. The dataset is built using an automated, scalable pipeline applied to in-the-wild internet videos from YouTube-based and Vimeo-based sources. Compared to past datasets, we show that videos from FoleyBench have stronger coverage of sound categories from a taxonomy specifically designed for Foley sound. Each clip is further labeled with metadata capturing source complexity, UCS/AudioSet category, and video length, enabling fine-grained analysis of model performance and failure modes. We benchmark several state-of-the-art V2A models, evaluating them on audio quality, audio-video alignment, temporal synchronization, and audio-text consistency. Samples are available at: https://gclef-cmu.org/foleybench

Figures (4)

• Figure 1: Dataset construction pipeline. Raw videos are first collected from Creative Commons (CC) licensed sources. Scenes are detected, trimmed to start/end timestamps, and discarded if shorter than 8s. Content-based filtering is then done in two stages: (1) YAMNet removes clips with speech or music, and (2) Gemini discards clips where audio is not visually or causally grounded, yielding high-quality Foley clips.
• Figure 2: Prompt for video content filtration and categorization.
• Figure 3: Prompt for video content categorization into AudioSet and UCS categories.
• Figure 4: Comparison of UCS Category Distributions. The top histogram shows the distribution of the filtered VGGSound test set (after removing music and speech videos), which is heavily skewed towards categories like ANIMALS and BEEPS. The bottom histogram shows the more balanced and uniform distribution of FoleyBench, which provides better coverage across a wider range of Foley-relevant sound categories.