Speech2UnifiedExpressions: Synchronous Synthesis of Co-Speech Affective Face and Body Expressions from Affordable Inputs

TL;DR

A multimodal learning-based method to simultaneously synthesize co-speech facial expressions and upper-body gestures for digital characters using RGB video data captured using commodity cameras and an adversarial discriminator to enhance the plausibility of synthesis.

Abstract

We present a multimodal learning-based method to simultaneously synthesize co-speech facial expressions and upper-body gestures for digital characters using RGB video data captured using commodity cameras. Our approach learns from sparse face landmarks and upper-body joints, estimated directly from video data, to generate plausible emotive character motions. Given a speech audio waveform and a token sequence of the speaker's face landmark motion and body-joint motion computed from a video, our method synthesizes the motion sequences for the speaker's face landmarks and body joints to match the content and the affect of the speech. We design a generator consisting of a set of encoders to transform all the inputs into a multimodal embedding space capturing their correlations, followed by a pair of decoders to synthesize the desired face and pose motions. To enhance the plausibility of synthesis, we use an adversarial discriminator that learns to differentiate between the face and pose motions computed from the original videos and our synthesized motions based on their affective expressions. To evaluate our approach, we extend the TED Gesture Dataset to include view-normalized, co-speech face landmarks in addition to body gestures. We demonstrate the performance of our method through thorough quantitative and qualitative experiments on multiple evaluation metrics and via a user study. We observe that our method results in low reconstruction error and produces synthesized samples with diverse facial expressions and body gestures for digital characters.

Figures (6)

• Figure 1: Network architecture for synchronous synthesis of co-speech face and pose expressions. Our generator encodes all the inputs: the speech audio, the corresponding test transcript, the speaker ID, the seed 3D face landmarks, and the seed 3D poses into a multimodal embedding space. It decodes variables from this space to produce the synchronized sequences of co-speech 3D face landmarks and poses. Our discriminator classifies these synthesized sequences and the corresponding ground truths (3D motions of the original speakers), computed directly from the videos, into two different classes based both on their plausibility and their synchronous expressions. To obtain our rendered 3D character motions, we combine the outputs of our generator with our phoneme predictor network and map them to 3D meshes.
• Figure 2: Face and pose encoders and decoders. We show their architectures with the layer sizes denoted (details in Sec. \ref{['subsubsec:enc_aff_exp']}). Our architectures depend on the hierarchical anatomical component (AC) graphs for both faces and poses that efficiently learn their corresponding affect representations using spatial-temporal graph convolutions (green nodes and edges), 2D convolutions (teal blocks), 2D batch normalizations (pink blocks), and fully-connected layers (orange planes).
• Figure 3: Qualitative results. Snapshots from two of our synthesized samples showing the text transcript of the speech and the corresponding face and pose expressions (row 1). We also zoom in on the eyebrow (row 2) and lip (row 3) expressions for better visualization. We observe a smile, raised eyebrows, and stretched arms (left) for the word 'excited', and frowns on the eyebrows and lips (right) for the words 'very sorry'.
• Figure 4: Qualitative comparisons. For the same input speech, represented by the text transcript at the top, we compare the visual quality of our synthesized character motions with the original speaker motions and three of our ablated versions: one without synchronous face and pose synthesis, one without our anatomical component (AC) graphs for faces and poses, and one without our discriminator. We observe that our synthesized motions are visually the closest to the original speaker motions compared to the ablated versions. We elaborate on their visual qualities in Sec. \ref{['subsec:qualitative_comparisons']}.
• Figure 5: Distributions of the user study responses. Likert-scale response distributions to the two sets of motions rendered using the five different types of face landmark and pose data (Sec. \ref{['sec:user_study']}). We show the distributions of each of the five Likert-scale points for each type of motion as a percentage of the total responses across all the groups in each set.