EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation

TL;DR

EchoFoley tackles the limitations of video-grounded sound generation by introducing an event-centric, hierarchical control paradigm that explicitly models when, what, and how sounds occur in a video. It defines a symbolic sounding-event representation and builds EchoFoley-6k to support fine-grained instruction-guided audio synthesis, alongside evaluation metrics for controllability and perceptual quality. The authors further propose EchoVidia, a training-free agentic framework with slow–fast thinking that significantly improves temporal grounding, sound identity manipulation, and overall alignment with visual cues, outperforming prior VT2A baselines. Together, these contributions advance controllable, contextually coherent audiovisual generation and offer a scalable path toward omni-modal intelligences that can reason about complex audio-visual narratives. The work has practical implications for film post-production, content creation, and multimodal world modeling by enabling precise, rule-based sound design grounded in visual scenes and textual instructions.

Abstract

Sound effects build an essential layer of multimodal storytelling, shaping the emotional atmosphere and the narrative semantics of videos. Despite recent advancement in video-text-to-audio (VT2A), the current formulation faces three key limitations: First, an imbalance between visual and textual conditioning that leads to visual dominance; Second, the absence of a concrete definition for fine-grained controllable generation; Third, weak instruction understanding and following, as existing datasets rely on brief categorical tags. To address these limitations, we introduce EchoFoley, a new task designed for video-grounded sound generation with both event level local control and hierarchical semantic control. Our symbolic representation for sounding events specifies when, what, and how each sound is produced within a video or instruction, enabling fine-grained controls like sound generation, insertion, and editing. To support this task, we construct EchoFoley-6k, a large-scale, expert-curated benchmark containing over 6,000 video-instruction-annotation triplets. Building upon this foundation, we propose EchoVidia a sounding-event-centric agentic generation framework with slow-fast thinking strategy. Experiments show that EchoVidia surpasses recent VT2A models by 40.7% in controllability and 12.5% in perceptual quality.

Figures (8)

• Figure 1: Motivation of EchoFoley. In creative storytelling, sound shapes the story we perceive. Given a silent video, generating audio that matches this story-shaped imagination requires fine-grained control over how each sound is crafted and transformed over time. We formulate such task with event-centric hierarchical control (e.g., instance control, group control), and propose effective solution.
• Figure 2: Data Curation Pipeline of EchoFoley-6k
• Figure 3: Video Topic Distribution in EchoFoley-6k
• Figure 4: The relationship of Instruction Adherence ($x$-axis) plotted against Audio–Visual Coherence ($y$-axis) of models shows Visual Dominance Bias.
• Figure 5: Average controllability scores for different levels of control granularity, from macro (global) to micro (atomic).