Empowering Sign Language Communication: Integrating Sentiment and Semantics for Facial Expression Synthesis

TL;DR

Sign language production must convey both manual gestures and non-manual facial expressions that encode sentence semantics and speaker sentiment. The paper presents a sentiment-aware facial expression synthesis pipeline that builds a meaningful latent space with Generative Latent Optimization (GLO) and uses a Sampling Network I to map semantic and sentiment features to latent codes, which are decoded by a Residual Spatio-Temporal Graph Convolutional Network and rendered via image-to-image translation. A new metric, Fréchet Expression Distance (FED), is introduced to quantify the quality of generated expressions, and extensive experiments on How2Sign and PHOENIX14T demonstrate state-of-the-art results. The work advances sign-language technology by enabling expressive, emotion-aware facial gestures from text, potentially improving accessibility and communication for deaf and hard-of-hearing communities.

Abstract

Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker's emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speakers expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Frechet Gesture Distance (FGD) and propose a new metric called Frechet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions.

Figures (12)

• Figure 1: Facial expression synthesis for sign language. Given a sentence of written text, our method generates distinct non-manual gestures (a, b, c and d) according to the sentiment information used as input to the model. All expressions in the image were automatically generated by our method.
• Figure 2: Overview of our facial expressions synthesis approach. Our method is composed of two main components: (a) At first, we learn a representation space that contains all the knowledge about the facial expressions in an organized manner; (b) Then we learn how to sample from our space, considering essential aspects in a dialogue, such as emotional state and semantics.
• Figure 3: Face graph topology. (a) We use the $k$-NN search to determine $k$ connections of each vertex. The figure shows the added edges (black lines) of three vertices (red dots); (b) Spatial upsampling graph pyramid. We designed a sequence of graphs with growing levels of detail. For sake of clarity the $k$-NN edges are omitted.
• Figure 4: Graph Convolutional Decoder. Our decoder network is composed of blocks temporal and spatial upsampling, and graph convolutions layers. The architecture maps from a meaningful representation space to face moving landmarks.
• Figure 5: Sampling Network. Our sampling network is composed of fully connected layers. The architecture maps between the concatenated sentiment $F_e$ and semantic features $F_s$ to the learned representation space $z$.