PAVAS: Physics-Aware Video-to-Audio Synthesis

TL;DR

PAVAS tackles the misalignment between visual dynamics and acoustic realism in video-to-audio synthesis by incorporating explicit physics reasoning. It introduces a Physics Parameter Estimator to extract object mass and velocity from video and a Physics-Driven Audio Adapter to inject these cues into a latent diffusion-based audio generator via Δ-modulation. The approach is validated on VGGSound and a new VGG-Impact benchmark, with APCC measuring how well generated audio tracks kinetic energy changes, showing superior physical plausibility without sacrificing perceptual quality. Overall, PAVAS advances physically grounded V2A generation, enabling sounds that coherently reflect real-world object dynamics and interactions. This work also provides a new evaluation protocol for physical realism in V2A systems, facilitating future research in physics-aware audio synthesis.

Abstract

Recent advances in Video-to-Audio (V2A) generation have achieved impressive perceptual quality and temporal synchronization, yet most models remain appearance-driven, capturing visual-acoustic correlations without considering the physical factors that shape real-world sounds. We present Physics-Aware Video-to-Audio Synthesis (PAVAS), a method that incorporates physical reasoning into a latent diffusion-based V2A generation through the Physics-Driven Audio Adapter (Phy-Adapter). The adapter receives object-level physical parameters estimated by the Physical Parameter Estimator (PPE), which uses a Vision-Language Model (VLM) to infer the moving-object mass and a segmentation-based dynamic 3D reconstruction module to recover its motion trajectory for velocity computation. These physical cues enable the model to synthesize sounds that reflect underlying physical factors. To assess physical realism, we curate VGG-Impact, a benchmark focusing on object-object interactions, and introduce Audio-Physics Correlation Coefficient (APCC), an evaluation metric that measures consistency between physical and auditory attributes. Comprehensive experiments show that PAVAS produces physically plausible and perceptually coherent audio, outperforming existing V2A models in both quantitative and qualitative evaluations. Visit https://physics-aware-video-to-audio-synthesis.github.io for demo videos.

Figures (3)

• Figure 1: Physics-Aware Video-to-Audio Synthesis (PAVAS). [Top] Current V2A models often generate physically inconsistent audio. [Bottom] We estimate physics values (object-level mass and velocity) from an input video using Physics Parameter Estimator, which are explicitly integrated into a latent diffusion-based model using Phy-Adapter to generate a physically plausible audio.
• Figure 2: Overall pipeline of the proposed Physics-Aware Video-to-Audio Synthesis (PAVAS). Given an input video, the Physics Parameter Estimator (PPE) extracts object-level mass and velocity. These physics cues are encoded by the Physics-Driven Audio Adapter (Phy-Adapter) and injected into the latent diffusion model alongside multimodal conditions. $E_p$ stands for CLIP vision encoder for patch embeddings, $E_s$ for SyncFormer iashin2024synchformer vision encoder, $E_v$ for CLIP vision encoder for flattened visual tokens, $E_t$ for CLIP text encoder, and $E_a$ for VAE/STFT-based audio encoder. Dashed lines indicate conditioning pathways, and magenta highlights physics-based conditioning.
• Figure 3: Qualitative comparison of generated spectrograms. We visualize spectrograms from existing V2A models xing2024seeington2025taroviertola2025temporallyliu2024tellcheng2025mmaudio, our method, and the ground truth. Green dashed lines indicate spectral patterns temporally aligned with visual events in the video, and graphic icons denote audible objects or interactions present in the audio track. PAVAS produces spectral patterns that more closely align with these events, whereas other methods often generate components that are not well aligned with the visual dynamics.