STSA: Spatial-Temporal Semantic Alignment for Visual Dubbing

TL;DR

STSA tackles semantic ambiguity between spatial and temporal domains in audio-driven visual dubbing by introducing a dual-path alignment framework and a differentiable probabilistic heatmap guidance. The Consistent Information Learning (CIL) module maximizes mutual information across scales to align spatial-temporal semantics, while the heatmap guidance provides ambiguity-tolerant cues that enable end-to-end optimization. Empirical results on LRS2 and CMLR show STSA improves image quality and synthesis stability, with strong cross-domain generalization compared to state-of-the-art baselines. The work demonstrates that explicit spatial-temporal semantic alignment and differentiable guidance are key for more realistic and stable visual dubbing in practical applications.

Abstract

Existing audio-driven visual dubbing methods have achieved great success. Despite this, we observe that the semantic ambiguity between spatial and temporal domains significantly degrades the synthesis stability for the dynamic faces. We argue that aligning the semantic features from spatial and temporal domains is a promising approach to stabilizing facial motion. To achieve this, we propose a Spatial-Temporal Semantic Alignment (STSA) method, which introduces a dual-path alignment mechanism and a differentiable semantic representation. The former leverages a Consistent Information Learning (CIL) module to maximize the mutual information at multiple scales, thereby reducing the manifold differences between spatial and temporal domains. The latter utilizes probabilistic heatmap as ambiguity-tolerant guidance to avoid the abnormal dynamics of the synthesized faces caused by slight semantic jittering. Extensive experimental results demonstrate the superiority of the proposed STSA, especially in terms of image quality and synthesis stability. Pre-trained weights and inference code are available at https://github.com/SCAILab-USTC/STSA.

Figures (4)

• Figure 1: Insight of the proposed STSA. Given the target masked face, driving audio, and reference faces, STSA generates a target face synchronized with audio by aligning the multi-scale features of reference faces in both spatial and temporal domains. Functional Modules refer to neural networks used for encoding, decoding, guidance prediction, and face synthesis.
• Figure 2: Illustration of the proposed Spatial-Temporal Semantic Alignment (STSA) approach. STSA consists of guidance prediction and face synthesis. The information within the purple dashed box are utilized multiple times throughout the process. The red dashed arrow means supervisions for heatmaps.
• Figure 3: Qualitative results on LRS2. Please zoom in to get more details.
• Figure 4: We visualize the motion trajectories of the heatmap in this paper and the sketch in IP-LAP with the Lucas Kanade algorithm, as shown in (a). Furthermore, we plot the trajectories of the right mouth corner in the coordinate system in (b).