T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text

TL;DR

This paper tackles sign language production by introducing a two-stage framework that first encodes sign sequences into dynamic, variable-length discrete codes using DVQ-VAE and then autoregressively generates code sequences from text via T2S-GPT, complemented by a duration-aware decoding mechanism. The DVQ-VAE stage employs a dynamic encoder/decoder with an information-density-based downsampling strategy, plus a budget loss and a translation auxiliary loss to preserve semantics. The second stage uses a Code-Transformer and a Duration-Transformer to predict code indices and their durations, enabling accurate reconstruction of sign language sequences; the model achieves state-of-the-art back-translation metrics on PHOENIX14T. A large German sign language dataset, PHOENIX-News, is introduced to study the impact of training data size and to support broader SLP research, with results indicating that more data yields further gains. Overall, the work demonstrates that variable-length, information-density-aware representations combined with duration-aware autoregression can improve the quality and efficiency of text-to-sign language generation, with practical implications for accessible communication and sign language research.

Abstract

In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook. However, existing vector quantization (VQ) methods are fixed-length encodings, overlooking the uneven information density in sign language, which leads to under-encoding of important regions and over-encoding of unimportant regions. To address this issue, we propose a novel dynamic vector quantization (DVA-VAE) model that can dynamically adjust the encoding length based on the information density in sign language to achieve accurate and compact encoding. Then, a GPT-like model learns to generate code sequences and their corresponding durations from spoken language text. Extensive experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. To promote sign language research, we propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts.Experimental analysis on PHOENIX-News shows that the performance of our model can be further improved by increasing the size of the training data. Our project homepage is https://t2sgpt-demo.yinaoxiong.cn.

Figures (6)

• Figure 1: Comparison of fixed-length encoding and variable-length encoding.
• Figure 2: The overview of our proposed two-stage framework.
• Figure 3: Distribution of text length and video duration in the PHOENIX-News dataset.
• Figure 4: Qualitative results of our T2S-GPT model.
• Figure 5: Impact of dataset size on the performance of T2S-GPT.