Latent Behavior Diffusion for Sequential Reaction Generation in Dyadic Setting

TL;DR

This paper tackles the challenge of generating diverse and contextually appropriate listener facial reactions in dyadic interactions. It introduces Latent Behavior Diffusion, a two-stage, non-autoregressive architecture that combines a context-aware autoencoder with a latent diffusion generator operating in a compact latent space, enabling multiple plausible reactions from speaker behavior. The method defines a formal one-to-many mapping, trains with VQ-VAE-inspired losses, and uses a conditional diffusion process with PLMS sampling guided by semantic encodings to produce diverse, synchronized reactions. Evaluations on the REACT2024 dataset show state-of-the-art performance in diversity and synchrony, with ablations demonstrating the importance of denoising steps and PLMS order for balancing diversity and accuracy. This approach advances realistic, controllable dyadic reaction synthesis with potential applications in conversational agents and human-centered AI systems.

Abstract

The dyadic reaction generation task involves synthesizing responsive facial reactions that align closely with the behaviors of a conversational partner, enhancing the naturalness and effectiveness of human-like interaction simulations. This paper introduces a novel approach, the Latent Behavior Diffusion Model, comprising a context-aware autoencoder and a diffusion-based conditional generator that addresses the challenge of generating diverse and contextually relevant facial reactions from input speaker behaviors. The autoencoder compresses high-dimensional input features, capturing dynamic patterns in listener reactions while condensing complex input data into a concise latent representation, facilitating more expressive and contextually appropriate reaction synthesis. The diffusion-based conditional generator operates on the latent space generated by the autoencoder to predict realistic facial reactions in a non-autoregressive manner. This approach allows for generating diverse facial reactions that reflect subtle variations in conversational cues and emotional states. Experimental results demonstrate the effectiveness of our approach in achieving superior performance in dyadic reaction synthesis tasks compared to existing methods.

Figures (4)

• Figure 1: Overview of our proposed Latent Diffusion Model (LDM) for generating multiple reactions. During the training phase, the autoencoder (AE) is first trained to encode the time series of listener reactions through a reconstruction task. Concurrently, the LDM is trained to predict future targets based on speaker behaviors $\mathcal{C}$. During the sampling phase, the latent representation of the time series is first generated by the LDM and then passed as input to the decoder $\mathcal{D}_{r}(z_0)$ to obtain the future targets.
• Figure 2: Conditional Behavior Decoder architecture in Latent space.
• Figure 3: The qualitative results of the generated facial frames of multiple listeners. Our comparison between other baselines by 3D rendering translation. The bottom of the figure shows our model-generated variants of reaction that are expressed from Speaker ground truth. The time $t$ in second.
• Figure 4: Generated listener facial reactions from different speakers.