Drivable 3D Gaussian Avatars

TL;DR

D3GA introduces a light, composable framework for drivable 3D human avatars by embedding 3D Gaussian primitives inside tetrahedral cages, and applying cage-driven deformation gradients to the Gaussian covariances. The method supports layered avatar components (body, face, garments) with localized conditioning signals (poses, keypoints, embeddings) and uses a dedicated GaussianNet per part to predict cage corrections and colors, enabling real-time rendering with improved garment separation. It demonstrates superior PSNR/SSIM on multi-view datasets and robustness to varied body shapes and clothes, while maintaining compact parameter counts and flexibility for extension to other regions. The work highlights practical benefits for telepresence and immersive applications, while acknowledging limitations such as high-frequency detail and collision handling, and suggesting future directions toward relightable appearance models and forensics-aware safeguards.

Abstract

We present Drivable 3D Gaussian Avatars (D3GA), a multi-layered 3D controllable model for human bodies that utilizes 3D Gaussian primitives embedded into tetrahedral cages. The advantage of using cages compared to commonly employed linear blend skinning (LBS) is that primitives like 3D Gaussians are naturally re-oriented and their kernels are stretched via the deformation gradients of the encapsulating tetrahedron. Additional offsets are modeled for the tetrahedron vertices, effectively decoupling the low-dimensional driving poses from the extensive set of primitives to be rendered. This separation is achieved through the localized influence of each tetrahedron on 3D Gaussians, resulting in improved optimization. Using the cage-based deformation model, we introduce a compositional pipeline that decomposes an avatar into layers, such as garments, hands, or faces, improving the modeling of phenomena like garment sliding. These parts can be conditioned on different driving signals, such as keypoints for facial expressions or joint-angle vectors for garments and the body. Our experiments on two multi-view datasets with varied body shapes, clothes, and motions show higher-quality results. They surpass PSNR and SSIM metrics of other SOTA methods using the same data while offering greater flexibility and compactness.

Figures (16)

• Figure 1: Overview. D3GA uses 3D pose $\pmb{\phi}$, face embedding $\pmb{\kappa}$, viewpoint $\mathbf{d}_k$ and canonical cage $\mathbf{v}$ (as well as auto-decoded color features $\mathbf{h}_i$) to generate the final render $\mathbf{\bar{C}}$ and auxiliary segmentation render $\mathbf{\bar{P}}$. The inputs in the left are processed through three networks ($\mathbf{\Psi}_\mathrm{MLP}$, $\mathbf{\Pi}_\mathrm{MLP}$, $\mathbf{\Gamma}_\mathrm{MLP}$) per avatar part to generate cage displacements $\mathbf{\Delta v}$, Gaussians deformations $\mathbf{b}_i$, $\mathbf{q}_i$, $\mathbf{s}_i$ and color/oppacity $\mathbf{c}_i$, $o_i$ respectively. After cage deformations transform canonical Gaussians, they are rasterized into the final images according to Eq. \ref{['formula: splatting&volume rendering']}.
• Figure 2: D3GA uses a tetrahedral mesh for deformation modeling.
• Figure 3: Qualitative comparisons show that D3GA models facial expressions and garments better than other SOTA approaches. Especially regions with loose garments like skirts or sweatpants.
• Figure 4: ActorsHQ Isik2023TOG comprises challenging garments that contain high-frequency patterns. Our method despite its small size can capture it and performs the best in terms of PSNR and SSIM, ranking second only in terms of sharpness to AG Li2023AnimatableGL, which presents very sharp results due to the powerful StyleUNet wang2023styleavatar.
• Figure 5: D3GA enables motion transfer showing good generalizability while preserving each avatar's high-quality details.