SOAP: Style-Omniscient Animatable Portraits

TL;DR

SOAP addresses the challenge of creating fully animatable 3D facial avatars from a single portrait across diverse styles by combining a style-omniscient, multi-view diffusion model with an adaptive, topology-preserving FLAME deformation pipeline. It leverages a 24K-head dataset to train six-view image and normal diffusion priors, then iteratively deforms a parametric FLAME mesh with semantic and landmark constraints while performing topology correction and rig updates, followed by eyeball, teeth, and texture fitting. The approach yields high-quality, textured, rigged heads that support FACS-based animation and view-consistent rendering across realistic and stylized styles, outperforming both diffusion-based and parametric baselines in qualitative and quantitative evaluations. The method demonstrates practical value for rapid, style-diverse avatar creation and open research via public code and data release, enabling broader applications in games, media, and interactive virtual environments.

Abstract

Creating animatable 3D avatars from a single image remains challenging due to style limitations (realistic, cartoon, anime) and difficulties in handling accessories or hairstyles. While 3D diffusion models advance single-view reconstruction for general objects, outputs often lack animation controls or suffer from artifacts because of the domain gap. We propose SOAP, a style-omniscient framework to generate rigged, topology-consistent avatars from any portrait. Our method leverages a multiview diffusion model trained on 24K 3D heads with multiple styles and an adaptive optimization pipeline to deform the FLAME mesh while maintaining topology and rigging via differentiable rendering. The resulting textured avatars support FACS-based animation, integrate with eyeballs and teeth, and preserve details like braided hair or accessories. Extensive experiments demonstrate the superiority of our method over state-of-the-art techniques for both single-view head modeling and diffusion-based generation of Image-to-3D. Our code and data are publicly available for research purposes at https://github.com/TingtingLiao/soap.

Figures (24)

• Figure 1: Method overview. Given an input image $\mathrm{I}$, SOAP (a) generates six orthogonal RGB images $\mathcal{I}$ and normal images $\mathcal{N}$, then (b) deforms the FLAME mesh $\mathcal{F}(\bar{\Omega}, \kappa_{\mathrm{I}})$ to $\mathcal{F}({\Omega}^*, \kappa_{\mathrm{I}})$, and (c) fits eyeballs and teeth to the mesh and generates the texture map.
• Figure 2: 3D Head dataset. The idea is to train the diffusion module with only two extreme styles, i.e., realistic and anime (non-realistic), and generalize to unseen intermediate styles.
• Figure 3: Motivation for topology correction. The top and bottom rows show results with and without topology correction. In this example, the optimized mesh fails to reconstruct the geometric details of the hair and face without topology correction. Due to the significant deformation of hair starting from the FLAME scalp, there is a tendency for undesired twists and collapses, as highlighted in the red boxes.
• Figure 4: Template optimization losses. Illustrations of reconstruction, semantic, and landmark losses to template deformation.
• Figure 5: Ablation study. Impact of landmark loss, semantic loss, and remeshing on mesh reconstruction.