Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation

TL;DR

This work proposes a novel framework, Sign-D2C, that employs a conditional diffusion model to synthesize contextually smooth transition frames, enabling the seamless construction of continuous sign language sequences.

Abstract

Generating continuous sign language videos from discrete segments is challenging due to the need for smooth transitions that preserve natural flow and meaning. Traditional approaches that simply concatenate isolated signs often result in abrupt transitions, disrupting video coherence. To address this, we propose a novel framework, Sign-D2C, that employs a conditional diffusion model to synthesize contextually smooth transition frames, enabling the seamless construction of continuous sign language sequences. Our approach transforms the unsupervised problem of transition frame generation into a supervised training task by simulating the absence of transition frames through random masking of segments in long-duration sign videos. The model learns to predict these masked frames by denoising Gaussian noise, conditioned on the surrounding sign observations, allowing it to handle complex, unstructured transitions. During inference, we apply a linearly interpolating padding strategy that initializes missing frames through interpolation between boundary frames, providing a stable foundation for iterative refinement by the diffusion model. Extensive experiments on the PHOENIX14T, USTC-CSL100, and USTC-SLR500 datasets demonstrate the effectiveness of our method in producing continuous, natural sign language videos.

Figures (4)

• Figure 1: Task and key steps. Our work aims to generate continuous sign videos by creating transition poses between discrete segments. In training, random masking simulates missing transitions, and the model learns to recover these frames (steps 1-3). During inference, padding initializes missing transitions, which the model refines to generate smooth, coherent sequences (steps 4-6).
• Figure 2: Overview. The proposed framework for generating continuous sign language videos with smooth transitions between discrete segments. In the training phase (top), a long-duration sign video undergoes random masking to create gaps, simulating missing transitions. An encoder processes the video, and a conditional diffusion model denoises and predicts the masked segments based on observable frames. The recovered video is produced through a decoder, with both encoder and decoder pre-trained for effective representation. In the inference phase (bottom), discrete sign videos are connected using a linearly interpolating padding strategy to initialize missing transitions. Gaussian noise is applied, and the model refines these transitions, generating a continuous sign video with coherent flow.
• Figure 3: Visualization examples of generating 10-frame transition pose under 20-frame observations on PHOENIX14T. We compare our method with G2P-DDM and VQ-GCDM, attached the Ground Truth. Green: Transitions. Orange and Blue: Observations.
• Figure 4: Visualization examples on the USTC-SLR500, demonstrate the generation of transition poses according to two discrete sign segments. Green: Transitions. Orange and Blue: Observations.