Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video

Ciara Rowles, Varun Jampani, Simon Donné, Shimon Vainer, Julian Parker, Zach Evans

TL;DR

Foley Control tackles practical video-guided Foley generation by freezing a strong text-to-audio backbone and learning only a lightweight video–audio bridge fed by V-JEPA2 video embeddings. The bridge is implemented as cross-attention modules inserted after the model's text cross-attention, with token pooling and RoPE to achieve temporal grounding without retraining the audio prior. Evaluated on MovieGenBench with a curated dataset, it achieves competitive temporal and semantic alignment while using far fewer trainable parameters than fully multimodal systems, preserving prompt-driven controllability and modularity. The approach suggests a scalable path to efficient video-to-audio synthesis that can extend to other audio modalities and production settings.

Abstract

Foley Control is a lightweight approach to video-guided Foley that keeps pretrained single-modality models frozen and learns only a small cross-attention bridge between them. We connect V-JEPA2 video embeddings to a frozen Stable Audio Open DiT text-to-audio (T2A) model by inserting compact video cross-attention after the model's existing text cross-attention, so prompts set global semantics while video refines timing and local dynamics. The frozen backbones retain strong marginals (video; audio given text) and the bridge learns the audio-video dependency needed for synchronization -- without retraining the audio prior. To cut memory and stabilize training, we pool video tokens before conditioning. On curated video-audio benchmarks, Foley Control delivers competitive temporal and semantic alignment with far fewer trainable parameters than recent multi-modal systems, while preserving prompt-driven controllability and production-friendly modularity (swap/upgrade encoders or the T2A backbone without end-to-end retraining). Although we focus on Video-to-Foley, the same bridge design can potentially extend to other audio modalities (e.g., speech).

Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video

TL;DR

Foley Control tackles practical video-guided Foley generation by freezing a strong text-to-audio backbone and learning only a lightweight video–audio bridge fed by V-JEPA2 video embeddings. The bridge is implemented as cross-attention modules inserted after the model's text cross-attention, with token pooling and RoPE to achieve temporal grounding without retraining the audio prior. Evaluated on MovieGenBench with a curated dataset, it achieves competitive temporal and semantic alignment while using far fewer trainable parameters than fully multimodal systems, preserving prompt-driven controllability and modularity. The approach suggests a scalable path to efficient video-to-audio synthesis that can extend to other audio modalities and production settings.

Abstract

Foley Control is a lightweight approach to video-guided Foley that keeps pretrained single-modality models frozen and learns only a small cross-attention bridge between them. We connect V-JEPA2 video embeddings to a frozen Stable Audio Open DiT text-to-audio (T2A) model by inserting compact video cross-attention after the model's existing text cross-attention, so prompts set global semantics while video refines timing and local dynamics. The frozen backbones retain strong marginals (video; audio given text) and the bridge learns the audio-video dependency needed for synchronization -- without retraining the audio prior. To cut memory and stabilize training, we pool video tokens before conditioning. On curated video-audio benchmarks, Foley Control delivers competitive temporal and semantic alignment with far fewer trainable parameters than recent multi-modal systems, while preserving prompt-driven controllability and production-friendly modularity (swap/upgrade encoders or the T2A backbone without end-to-end retraining). Although we focus on Video-to-Foley, the same bridge design can potentially extend to other audio modalities (e.g., speech).
Paper Structure (34 sections, 2 figures, 2 tables)

This paper contains 34 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. FoleyCrafter.
  4. MMAudio.
  5. Stable Audio Open.
  6. V-JEPA2.
  7. Other related V2A / audiovisual models.
  8. Architectural Positioning of our approach
  9. Method
  10. Preliminaries
  11. Audio Latent Diffusion.
  12. Video Representation Learning.
  13. Problem Setup.
  14. Dataset Curation
  15. Framework Overview
  16. ...and 19 more sections

Figures (2)

  • Figure 1: Forward flow: frames $\rightarrow$ V– JEPA2 $\rightarrow$ adapter (video vCA), prompt $\rightarrow$ text encoder (Tx– CA), and noise (latent init) entering the DiT at the same level
  • Figure 2: Ablation trends across training steps for different pooling strategies, evaluated with the KL-PANNs metric (lower is better) on the Kinetics-700 validation subset without text guidance. All runs were trained on a fixed 30% subset of the curated training data.