MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation

TL;DR

MaDiS introduces a masked diffusion language model (MDLM) for sign language generation to overcome the context and efficiency limits of autoregressive SLG. It employs a tri-level cross-modal pretraining strategy spanning token, latent, and 3D physical spaces, plus a novel Unmasking with Temporal Checkpoints (UTC) and a Mixture-of-Parts (MoP) embedding layer to speed training and improve sign fidelity. The approach achieves state-of-the-art performance on CSL-Daily, Phoenix-2014T, and How2Sign across DTW-JPE, SiBLEU, and SiCLIP while reducing inference latency by about 30%. These results demonstrate the value of bidirectional MDLMs and multi-space pretraining for grounded, efficient sign language generation, with code and models to be released on the project page.

Abstract

Sign language generation (SLG) aims to translate written texts into expressive sign motions, bridging communication barriers for the Deaf and Hard-of-Hearing communities. Recent studies formulate SLG within the language modeling framework using autoregressive language models, which suffer from unidirectional context modeling and slow token-by-token inference. To address these limitations, we present MaDiS, a masked-diffusion-based language model for SLG that captures bidirectional dependencies and supports efficient parallel multi-token generation. We further introduce a tri-level cross-modal pretraining scheme that jointly learns from token-, latent-, and 3D physical-space objectives, leading to richer and more grounded sign representations. To accelerate model convergence in the fine-tuning stage, we design a novel unmasking strategy with temporal checkpoints, reducing the combinatorial complexity of unmasking orders by over $10^{41}$ times. In addition, a mixture-of-parts embedding layer is developed to effectively fuse information stored in different part-wise sign tokens through learnable gates and well-optimized codebooks. Extensive experiments on CSL-Daily, Phoenix-2014T, and How2Sign demonstrate that MaDiS achieves superior performance across multiple metrics, including DTW error and two newly introduced metrics, SiBLEU and SiCLIP, while reducing inference latency by nearly 30%. Code and models will be released on our project page.

Figures (7)

• Figure 1: We propose MaDiS, a novel sign language generation approach built upon masked diffusion language models (MDLMs) llada. (a) A sign tokenizer discretizes continuous sign motions into part-wise tokens zuo2025soke. (b) Conventional autoregressive language models generate tokens in a left-to-right manner, limiting utilization of contexts and inference efficiency. (c) The emerging MDLMs model token distributions with bidirectional contexts and enable parallel multi-token sampling during inference. (d) MaDiS achieves SOTA performance across multiple benchmarks duarte2021how2signcsl-daily2014T while reducing inference latency by nearly 30%.
• Figure 2: MaDiS is built upon the emerging masked diffusion language model (MDLM), implemented by modifying a standard decoder-only LLM with bidirectional attention. The MDLM is first pretrained with three objectives ($\mathcal{L}_{tok}^{text} + \mathcal{L}_{tok}^{sign}$, $\mathcal{L}_{lat}$, $\mathcal{L}_{phy}$) from the token, latent, and 3D physical spaces, respectively. We then fine-tune the model conditioned on text inputs using the proposed temporal-checkpoint unmasking strategy and a dedicated mixture-of-parts sign embedding layer.
• Figure 3: (a) Vanilla MDLMs adopt an unconstrained confidence-based unmasking strategy llada. (b) We insert temporal checkpoints at noise levels $0.75$ and $0.5$ to reduce the complexity of unmasking orders. (c) The original unmasking strategy allows arbitrary unmasking orders (about $10^{123}$ for generating 100 tokens), while our method reduces this complexity by over $10^{41}$ times.
• Figure 4: (a) A typical sign embedding layer averages part-wise token embeddings zuo2025soke. (b) The proposed mixture-of-parts embedding layer leverages optimized VAE codebooks and uses a learnable gate to control contributions from different body parts.
• Figure 5: Qualitative comparisons of generated signs between our proposed method, MaDiS, with the SOTA method, SOKE zuo2025soke, on the test sets of CSL-Daily (left), Phoenix-2014T (middle), and How2Sign (right).