ReactDiff: Fundamental Multiple Appropriate Facial Reaction Diffusion Model

TL;DR

ReactDiff tackles online generation of multiple appropriate facial reactions in dyadic interactions by embedding two priors—temporal facial behavioral kinematics and facial action unit dependencies—into a diffusion-based framework. It generates multiple short AFR segments per time window, conditioned on speaker behavior and history, using a classifier-free, SDE-based sampling strategy. Training optimizes diffusion denoising with FBK and FAC losses, leveraging 3DMM coefficients and PIRender, and conditioning via adaptive normalization and cross-attention. On REACT2024, ReactDiff achieves state-of-the-art diversity, realism, and appropriateness, with ablations confirming the benefits of temporal plumbing, AU constraints, multi-modal conditioning, and stochastic sampling for natural, coherent reactions in real-time.

Abstract

The automatic generation of diverse and human-like facial reactions in dyadic dialogue remains a critical challenge for human-computer interaction systems. Existing methods fail to model the stochasticity and dynamics inherent in real human reactions. To address this, we propose ReactDiff, a novel temporal diffusion framework for generating diverse facial reactions that are appropriate for responding to any given dialogue context. Our key insight is that plausible human reactions demonstrate smoothness, and coherence over time, and conform to constraints imposed by human facial anatomy. To achieve this, ReactDiff incorporates two vital priors (spatio-temporal facial kinematics) into the diffusion process: i) temporal facial behavioral kinematics and ii) facial action unit dependencies. These two constraints guide the model toward realistic human reaction manifolds, avoiding visually unrealistic jitters, unstable transitions, unnatural expressions, and other artifacts. Extensive experiments on the REACT2024 dataset demonstrate that our approach not only achieves state-of-the-art reaction quality but also excels in diversity and reaction appropriateness.

Figures (8)

• Figure 1: Demonstration of diverse reactions generated by ReactDiff and Limitations of standard diffusion model for online facial reaction prediction.
• Figure 2: Overview of the proposed ReactDiff model. Left: the training stage of ReactDiff, wherein ReactDiff is learned to denoise 3D listener reaction sequence with given conditions and two constraints. Right: the inference stage of ReactDiff, involving the sampling of reaction sequences through multiple reverse diffusion steps.
• Figure 3: Illustration of three types of facial AU relationships.
• Figure 4: Qualitative Results on the REACT2024 test set. Each approach generates reaction sequences online based on a given sequence of speaker visual-audio behavior. Diversity in reactions is emphasized using red boxes, segments displaying a slow change speed are marked with blue boxes, while those with a rapid change speed are highlighted in orange boxes. Frames showing unnatural facial expressions or distortions are indicated by yellow boxes.
• Figure 5: Comparison of reactions from model without (w/o) the human temporal facial behavioral kinematics constraint $\phi_{\text{FBK}}(\cdot)$ and those from model with (w/) $\phi_{\text{FBK}}(\cdot)$.