SignSplat: Rendering Sign Language via Gaussian Splatting

TL;DR

SignSplat tackles the challenge of rendering expressive sign language by combining SMPL-X driven 3D reconstruction with a differentiable Gaussian splatting renderer anchored to the mesh. It introduces pose-aware Gaussian parameter prediction via a lightweight neural module, plus adaptive density control and regularization to stabilize sequence-level optimization. A novel sign stitching mechanism interleaves gloss-based signs into smooth SMPL-X trajectories, enabling high-quality, continuous renderings from limited views. The approach achieves state-of-the-art results on benchmark datasets and delivers superior rendering of complex hand articulation and facial expressions, with practical implications for accessible sign-language visualization and animation.

Abstract

State-of-the-art approaches for conditional human body rendering via Gaussian splatting typically focus on simple body motions captured from many views. This is often in the context of dancing or walking. However, for more complex use cases, such as sign language, we care less about large body motion and more about subtle and complex motions of the hands and face. The problems of building high fidelity models are compounded by the complexity of capturing multi-view data of sign. The solution is to make better use of sequence data, ensuring that we can overcome the limited information from only a few views by exploiting temporal variability. Nevertheless, learning from sequence-level data requires extremely accurate and consistent model fitting to ensure that appearance is consistent across complex motions. We focus on how to achieve this, constraining mesh parameters to build an accurate Gaussian splatting framework from few views capable of modelling subtle human motion. We leverage regularization techniques on the Gaussian parameters to mitigate overfitting and rendering artifacts. Additionally, we propose a new adaptive control method to densify Gaussians and prune splat points on the mesh surface. To demonstrate the accuracy of our approach, we render novel sequences of sign language video, building on neural machine translation approaches to sign stitching. On benchmark datasets, our approach achieves state-of-the-art performance; and on highly articulated and complex sign language motion, we significantly outperform competing approaches.

Figures (5)

• Figure 1: Overview: The input is a set of images. First, we obtain SMPL-X mesh reconstruction to obtain an initial human mesh. Then, we iteratively optimize Gaussian parameters such as opacity, rotation, spherical harmonics coefficients, and scale, along with mesh displacements to refine the representation across all images. During this optimization process, we adaptively densify and prune points on the mesh surface, jointly regularizing both the mesh and Gaussians.
• Figure 2: Improvements on novel views (a) and novel poses (b).
• Figure 3: Qualitative comparison of rendering results with the state-of-the-art on sign-language sequence.
• Figure 4: Qualitative comparison of rendering results with the state-of-the-art on the X-Humans dataset.
• Figure 5: Comparison of sign gloss interpolation between a skeleton-based GAN approach and our rendering method.