Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation

TL;DR

ACTalker addresses the challenge of multi-signal control in talking-head generation by integrating audio and visual facial-motion signals without conflicts. It introduces a Parallel-Control Mamba Layer (PCM) composed of multiple Mask-SSM branches and a mask-drop strategy to localize each driving signal to its target facial region, guided by a gating mechanism for flexible single- or multi-signal operation. The approach inserts these components into a Stable Video Diffusion backbone, employing flattened spatiotemporal features $z \in \mathbb{R}^{b \times (f \times h \times w) \times c}$ and identity conditioning $\mathbf{e}_{id}$ to preserve subject identity. Experiments across multiple datasets show improved audio-visual synchronization, visual quality, and expression fidelity, outperforming single-signal baselines and existing multi-signal approaches, with ablations confirming the effectiveness of PCM, Mask-SSM, and mask-drop in reducing control conflicts.

Abstract

Talking head synthesis is vital for virtual avatars and human-computer interaction. However, most existing methods are typically limited to accepting control from a single primary modality, restricting their practical utility. To this end, we introduce \textbf{ACTalker}, an end-to-end video diffusion framework that supports both multi-signals control and single-signal control for talking head video generation. For multiple control, we design a parallel mamba structure with multiple branches, each utilizing a separate driving signal to control specific facial regions. A gate mechanism is applied across all branches, providing flexible control over video generation. To ensure natural coordination of the controlled video both temporally and spatially, we employ the mamba structure, which enables driving signals to manipulate feature tokens across both dimensions in each branch. Additionally, we introduce a mask-drop strategy that allows each driving signal to independently control its corresponding facial region within the mamba structure, preventing control conflicts. Experimental results demonstrate that our method produces natural-looking facial videos driven by diverse signals and that the mamba layer seamlessly integrates multiple driving modalities without conflict. The project website can be found at https://harlanhong.github.io/publications/actalker/index.html.

Figures (12)

• Figure 1: In this work, we aim to develop a framework that not only generates videos driven by multiple signals without causing control conflicts in the facial region (first three rows) but also supports video generation driven by a single signal (last two rows).
• Figure 2: Illustration of our ACTalker framework. ACTalker takes multiple signals inputs (i.e., audio and visual facial motion) to drive the generation of talking head videos. In addition to the standard layers (e.g., spatial convolution, temporal convolution, spatial attention, and temporal attention) in the stable video diffusion model, we introduce a parallel-control mamba layer to harness the power of multiple signals control. Audio and facial motion signals are fed into this parallel-control mamba layer, along with their corresponding masks, which indicates the regions to focus on for manipulation.
• Figure 3: Illustration of parallel-control mamba layer. There are two parallel branches in this layer, one for audio control and the other is for expression control. We utilize a gate in each branch to control the accessing of control signal during training. During inference, we can manually modify the statue of gates to enable single signal control or multiple signals control.
• Figure 4: The illustrating of the Mask-SSM in audio branch of parallel-control mamba layer. The visual branch is the same but replace with the motion embedding and motion mask
• Figure 5: Comparison of different methods for audio-driven talking head generation. Our method can produce more natural and accurate lip-synced videos. Due to the page limitation, the results of SadTalker zhang2022sadtalker and Hallo xu2024hallo are reported in Supplementary Material