Efficient Listener: Dyadic Facial Motion Synthesis via Action Diffusion

TL;DR

This work introduces Facial Action Diffusion (FAD) and the Efficient Listener Network (ELNet) to synthesize realistic listener facial motions in dyadic conversations without relying on computationally expensive 3DMM coefficient extraction. The framework processes speaker visual and audio inputs in clip-based batches, using a conditional denoising diffusion process to map multimodal perception to listener motion represented in 3DMM space. Key contributions include a lightweight diffusion-based generator, a 1D U-Net backbone optimized for latent-space diffusion, and extensive experiments showing substantial speedups (up to 99% time reduction) while maintaining competitive accuracy and realism on ViCo and L2L datasets. The approach enables real-time, end-to-end listener motion generation with strong performance, offering practical impact for responsive, naturalistic human-computer interaction. Limitations point to generalization across languages and richer feedback modalities, suggesting future work on broader datasets and cultural adaptability.

Abstract

Generating realistic listener facial motions in dyadic conversations remains challenging due to the high-dimensional action space and temporal dependency requirements. Existing approaches usually consider extracting 3D Morphable Model (3DMM) coefficients and modeling in the 3DMM space. However, this makes the computational speed of the 3DMM a bottleneck, making it difficult to achieve real-time interactive responses. To tackle this problem, we propose Facial Action Diffusion (FAD), which introduces the diffusion methods from the field of image generation to achieve efficient facial action generation. We further build the Efficient Listener Network (ELNet) specially designed to accommodate both the visual and audio information of the speaker as input. Considering of FAD and ELNet, the proposed method learns effective listener facial motion representations and leads to improvements of performance over the state-of-the-art methods while reducing 99% computational time.

Figures (5)

• Figure 1: Facial Action Diffusion in the forward process. We treat the visual and auditory perception of the speaker as conditional probability parameters to guide the listener's facial motion synthesis through a denoising process. Although our initial setting is not a Gaussian noise image, we use it as a metaphorical description of our noise signal.
• Figure 2: The overview of the proposed method. We use multimodal data, including visual and auditory perception, as input to predict the listener facial feedback. We process the input at the granularity of clips as the smallest unit to achieve efficient handling of videos of varying lengths. Simultaneously, we innovatively regard the multimodal data as conditional probabilities for the diffusion noise addition and denoising processes, guiding the appropriate listener facial motions.
• Figure 3: The input video and audio are processed in clips. We predict the future listener facial motions using batch processing.
• Figure 4: The meta parameter studies results present the metrics for different numbers of iterations $K$ during the inference process. We tested with $K = 1, 5, 10$ on the ViCo dataset. We select L2 and SI as metrics for accuracy and diversity, respectively.
• Figure 5: We highlight all the contribution feature visualization results. The above shows the comparison results of the ResNet-18 in our framework and the parameters pretrained on ImageNet. We extracted a frame of the speaker from the ViCo dataset zhou2022responsive as an example.