ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactions

TL;DR

This paper reformulates the dyadic interaction task as an extrapolation or prediction problem, and proposes an novel framework (called ReactFace) to generate multiple different but appropriate facial reactions from a speaker behaviour rather than merely replicating the corresponding listener facial behaviours.

Abstract

In dyadic interaction, predicting the listener's facial reactions is challenging as different reactions could be appropriate in response to the same speaker's behaviour. Previous approaches predominantly treated this task as an interpolation or fitting problem, emphasizing deterministic outcomes but ignoring the diversity and uncertainty of human facial reactions. Furthermore, these methods often failed to model short-range and long-range dependencies within the interaction context, leading to issues in the synchrony and appropriateness of the generated facial reactions. To address these limitations, this paper reformulates the task as an extrapolation or prediction problem, and proposes an novel framework (called ReactFace) to generate multiple different but appropriate facial reactions from a speaker behaviour rather than merely replicating the corresponding listener facial behaviours. Our ReactFace generates multiple different but appropriate photo-realistic human facial reactions by: (i) learning an appropriate facial reaction distribution representing multiple different but appropriate facial reactions; and (ii) synchronizing the generated facial reactions with the speaker verbal and non-verbal behaviours at each time stamp, resulting in realistic 2D facial reaction sequences. Experimental results demonstrate the effectiveness of our approach in generating multiple diverse, synchronized, and appropriate facial reactions from each speaker's behaviour. The quality of the generated facial reactions is intimately tied to the speaker's speech and facial expressions, achieved through our novel speaker-listener interaction modules. Our code is made publicly available at \url{https://github.com/lingjivoo/ReactFace}.

Figures (10)

• Figure 1: Comparison between our ReactFace and facial reaction generation methods proposed by Ng. et al ng2022learning and Zhou et al. zhou2022responsive. The uppermost two rows delineate the auditory, and visual behaviours of a speaker's behavioural clip. The subsequent two rows display a pair of facial reaction sequences generated by ng2022learning, in response to the speaker's input. The fifth and sixth rows show a pair of facial reaction sequences generated by zhou2022responsive. Finally, the last two rows present diverse, appropriate and synchronised facial reactions generated by our ReactFace.
• Figure 2: The pipeline of the proposed ReactFace model.
• Figure 3: Illustration of: (a) multi-modal speaker behaviour encoding and alignment (MSBEA) module; (b) appropriate facial reaction generation (AFRG) module; and (c) speaker-listener behaviour synchronisation (SLBS) module.
• Figure 4: Illustration of visual interaction and modality interaction
• Figure 5: Comparison of: (a) point-to-point (P2P) audio-visual alignment proposed in Faceformer fan2022faceformer and (b) our proposed modality interaction. P2P audio-visual alignment primarily establishes corrections solely between a given visual frame and its corresponding audio frames at the identical timestamp (achieved through the assignment of zero attention weights) while effectively precluding the visual frame from attending to any other frames (achieved through the assignment of negative infinity attention weights). Our proposed modality interaction mechanism establishes both long-range and short-range relationships between a visual frame at a specific timestamp and audio frames occurring at both the current and previous time instances. This approach significantly enriches the capacity to capture intermodal dependencies and contextual nuances across disparate frames within the audio-visual data stream.