Modeling Intensification for Sign Language Generation: A Computational Approach

TL;DR

This paper aims to improve the prosody in generated sign languages by modeling intensification in a data-driven manner, and presents different strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations.

Abstract

End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial variations leads to poor-quality generated presentations that confuse human interpreters. In this paper, we aim to improve the prosody in generated sign languages by modeling intensification in a data-driven manner. We present different strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations. To employ our strategies, we first annotate a subset of the benchmark PHOENIX-14T, a German Sign Language dataset, with different levels of intensification. We then use a supervised intensity tagger to extend the annotated dataset and obtain labels for the remaining portion of it. This enhanced dataset is then used to train state-of-the-art transformer models for sign language generation. We find that our efforts in intensification modeling yield better results when evaluated with automatic metrics. Human evaluation also indicates a higher preference of the videos generated using our model.

Figures (5)

• Figure 1: In sign language, modifiers are represented spatially and temporally. Here, two signers from PHOENIX-14T manually sign German "less clouds", and "very cloudy". Both of these signs have the same gloss representation: WOLKE (cloud in German). They are figuratively the same sign, but the duration, repetition, temporal pauses, and continuations determine the exact meaning. This information is lost during sign language translation and evaluation.
• Figure 2: This figure shows an example annotation. German transcript text and gloss are provided as context along with their English translations. Each English gloss in the sentence are tagged with 0, 1, 2, corresponding to the degree of intensification.
• Figure 3: This figure shows the architecture of the Dynamic Selection model. The overall architecture is similar to the Progressive Transformer, except having two Encoders to select between two different types of strategies. MLP layer is the decisive step on selecting the strategy from the encoders. Dynamic model uses a weighted mixture of the decoder outputs (represented with a gradient of blue and red). Dynamic$_{hard}$ uses an argmax to pick a source.
• Figure 4: This figure illustrates the comparison between baseline and the intensification-enhanced model. Gloss annotations are linked to their corresponding frames. Here, ground truth skeleton uses wider movements due to the "heavy" modifier, and the intensification-enhanced outputs replicate the phenomena better than baseline.
• Figure 5: Human evaluation results for the generated skeletons.