Text-Driven Diffusion Model for Sign Language Production

TL;DR

This work tackles end-to-end sign language production by directly generating pose sequences from text using a Text-driven Diffusion Model (TDM). The model conditions a diffusion-based denoiser on text-derived semantics via a text encoder and an MLP, optimizing pose fidelity with joint and bone orientation losses: $\mathcal{L}_{joint}$ and $\mathcal{L}_{bone}$, combined as $\mathcal{L} = \mathcal{L}_{joint} + \lambda \mathcal{L}_{bone}$ with $\lambda = 0.1$. The methodology defines a forward process $p_t = \gamma_t p_0 + \sigma_t \epsilon$ with $\gamma_t^2 + \sigma_t^2 = 1$, and a reverse denoising stage $p_0' = \mathcal{D}(p_t, g)$, followed by DDIM-style iterative refinement. On PHOENIX14T, the approach achieves a BLEU-1 score of $20.17$ and strong DTW performance, earning second place and demonstrating promising direct text-to-pose synthesis for sign language with potential real-time applicability and broader accessibility.

Abstract

We introduce the hfut-lmc team's solution to the SLRTP Sign Production Challenge. The challenge aims to generate semantically aligned sign language pose sequences from text inputs. To this end, we propose a Text-driven Diffusion Model (TDM) framework. During the training phase, TDM utilizes an encoder to encode text sequences and incorporates them into the diffusion model as conditional input to generate sign pose sequences. To guarantee the high quality and accuracy of the generated pose sequences, we utilize two key loss functions. The joint loss function L_{joint} is used to precisely measure and minimize the differences between the joint positions of the generated pose sequences and those of the ground truth. Similarly, the bone orientation loss function L_{bone} is instrumental in ensuring that the orientation of the bones in the generated poses aligns with the actual, correct orientations. In the inference stage, the TDM framework takes on a different yet equally important task. It starts with noisy sequences and, under the strict constraints of the text conditions, gradually refines and generates semantically consistent sign language pose sequences. Our carefully designed framework performs well on the sign language production task, and our solution achieves a BLEU-1 score of 20.17, placing second in the challenge.

Figures (7)

• Figure 1: The task of generating sign language pose sequences directly from text extremely daunting challenges. Classic pipelines for Sign Language Production: text-to-gloss, gloss-to-pose and pose-to-real video. In this work, we go beyond the commonly used pipeline and focus on the T2P task of generating sign pose sequences directly from text.
• Figure 2: Overall framework of our method - TDM. It consists of a text encoder, a denoiser, and a MLP. The text encoder is primarily designed to capture the global semantics of the text sequence. This global semantics is integrated with the time step that has been processed by a MLP to form the relevant condition $g$. We add noise in $t$ steps to the target pose sequence $p_0$ to obtain $p_t$. Then, we send $p_t$ along with the relevant condition $g$ to the denoiser $\mathcal{D}$ to recover the target pose sequence $p_{0}'$ that is free from noise contamination. For the optimization, we employed two main loss functions. The $\mathcal{L}_{joint}$ loss function is primarily used to constrain the joint coordinates, while the $\mathcal{L}_{bone}$ loss function is mainly applied to constrain the direction of bone movement.
• Figure 3: Detailed implementation details of our denoiser $\mathcal{D}$.
• Figure 4: Parent-child node partition relationship on the PHOENIX14T dataset.
• Figure 5: Visualization examples of generated pose sequences on PHOENIX14T. We compare TDM with ground-truth.