Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization

TL;DR

This paper tackles sign language video generation from spoken language text by introducing SignViP, which leverages multiple fine-grained conditions (dense poses and 3D hands) via a discrete multi-condition token space. The framework decouples translation from generation through three components: a Sign Video Diffusion Model guided by continuous embeddings, an FSQ Autoencoder that compresses embeddings into discrete tokens, and a Multi-Condition Token Translator that converts text to tokens. Experimental results show SignViP achieves state-of-the-art performance across video quality, temporal coherence, and semantic fidelity on two sign-language datasets, with strong ablations validating the necessity of each component. The approach enables more natural, expressive, and signer-identifiable sign language videos, with publicly available code for reproducibility.

Abstract

Sign Language Video Generation (SLVG) seeks to generate identity-preserving sign language videos from spoken language texts. Existing methods primarily rely on the single coarse condition (\eg, skeleton sequences) as the intermediary to bridge the translation model and the video generation model, which limits both the naturalness and expressiveness of the generated videos. To overcome these limitations, we propose SignViP, a novel SLVG framework that incorporates multiple fine-grained conditions for improved generation fidelity. Rather than directly translating error-prone high-dimensional conditions, SignViP adopts a discrete tokenization paradigm to integrate and represent fine-grained conditions (\ie, fine-grained poses and 3D hands). SignViP contains three core components. (1) Sign Video Diffusion Model is jointly trained with a multi-condition encoder to learn continuous embeddings that encapsulate fine-grained motion and appearance. (2) Finite Scalar Quantization (FSQ) Autoencoder is further trained to compress and quantize these embeddings into discrete tokens for compact representation of the conditions. (3) Multi-Condition Token Translator is trained to translate spoken language text to discrete multi-condition tokens. During inference, Multi-Condition Token Translator first translates the spoken language text into discrete multi-condition tokens. These tokens are then decoded to continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide video generation. Experimental results show that SignViP achieves state-of-the-art performance across metrics, including video quality, temporal coherence, and semantic fidelity. The code is available at https://github.com/umnooob/signvip/.

Figures (7)

• Figure 1: (1) The illustration of SLVG task. (2) The pipeline comparison between existing SLVG methods (i.e., single-condition method and multi-condition method) and our SignViP. (3) Single-condition methods struggle to accurately capture the naturalness and expressiveness of sign language videos. (4) Multi-condition methods are prone to translation errors for fine-grained conditions.
• Figure 2: Framework of our SignViP for sign language video generation (SLVG). (1) The spoken language text is translated into the multi-condition tokens by Multi-Condition Token Translator. (2) These tokens are decoded by FSQ Autoencoder into multi-condition embeddings, which are equivalent to the embeddings of multiple fine-grained conditions (i.e., fine-grained poses and 3D hands) generated by a multi-condition encoder. (3) The embeddings are injected into Sign Video Diffusion Model to guide the generation of sign language videos.
• Figure 3: (a) Qualitative comparison on RWTH-2014T dataset. (b) Visual examples illustrating our SignViP's capability for identity generalization. (c) Effect of quantization. "R.(V)" and "R.(P)" mean ROUGE metrics of video and pose back-translation, as shown in Table \ref{['tab:back_translation_compare']} and Table \ref{['tab:pose_back_translation_compare']}, respectively. "FVD" evaluates the protocol described in Table \ref{['tab:video_quality']}.
• Figure 4: (a) Effect of compression rate. (b) Effect of condition augmentation probability. Note that "FVD" evaluates the protocol described in Table \ref{['tab:generative_capability']}. (c) Effect of sampling rate for the scheduled sampling strategy. (d) Codebook usage comparison between FSQ and VQ. (e) Multi-conditional reconstruction loss comparison between FSQ and VQ.
• Figure 5: (a) Effect of spatial perturbation for the pose back-translation model. (b) Effect of temporal perturbation for the pose back-translation model. (c) Effect of spatial perturbation for the video back-translation model. (d) Effect of temporal perturbation for the video back-translation model.