T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis

TL;DR

T-Foley introduces a waveform-domain diffusion model for Foley sound synthesis with explicit temporal control. By conditioning on a sound class and a temporal event feature, and by employing Block-FiLM to efficiently fuse temporal information, it achieves accurate timing and timbre without a pretrained vocoder. Objective metrics (FAD, IS, E-L1) and subjective MOS evaluations show improvements over non-temporal baselines, with Case Study results illustrating practical, human-like vocal mimicry for temporal guidance. The approach enables realistic, time-aligned Foley generation suitable for interactive media workflows and complex sound design tasks.

Abstract

Foley sound, audio content inserted synchronously with videos, plays a critical role in the user experience of multimedia content. Recently, there has been active research in Foley sound synthesis, leveraging the advancements in deep generative models. However, such works mainly focus on replicating a single sound class or a textual sound description, neglecting temporal information, which is crucial in the practical applications of Foley sound. We present T-Foley, a Temporal-event-guided waveform generation model for Foley sound synthesis. T-Foley generates high-quality audio using two conditions: the sound class and temporal event feature. For temporal conditioning, we devise a temporal event feature and a novel conditioning technique named Block-FiLM. T-Foley achieves superior performance in both objective and subjective evaluation metrics and generates Foley sound well-synchronized with the temporal events. Additionally, we showcase T-Foley's practical applications, particularly in scenarios involving vocal mimicry for temporal event control. We show the demo on our companion website.

Figures (5)

• Figure 1: Temporal-event-guided Foley sound synthesis.
• Figure 2: (a) Overall architecture of the proposed model. ($\mathbf{c}$: sound class, $\sigma$: diffusion timestep, $T$: temporal event feature) (b) A detailed structure of a Down/Up sampling block. Each Down block includes strided convolutional layer at first while Up blocks exploit the transposed one. ($\mathbf{h_{in}}/\mathbf{h_{out}}$: latent features) (c) Comparison of FiLM, TFiLM, and the proposed BFiLM. ($Y$: conditioning input, $X$: input activation)
• Figure 3: Tradeoff between performance(E-L1, FAD-P) and efficiency(inference time) among different number of blocks.
• Figure 4: The first row shows the control sounds used to extract the target event feature. Subsequent rows show three classes of Foley sounds generated with different conditioning blocks (FiLM, TFiLM, and BFiLM), all represented as mel-spectrograms.
• Figure 5: (a) Comparing manually synthesized consecutive gunshot sounds with sounds generated through temporal event feature. (b) Generated sounds with the original temporal event features and those with a reduced gain by 10.