ToonifyGB: StyleGAN-based Gaussian Blendshapes for 3D Stylized Head Avatars

TL;DR

ToonifyGB addresses the challenge of generating diverse stylized 3D head avatars from monocular video by coupling a stylized video generation stage with a Gaussian blendshape-based 3D avatar synthesis stage. It introduces architectural refinements to StyleGANEX to produce stable, unaligned-frame stylized videos, which feed into a Stage 2 that learns a neutral head model and a set of expression Gaussians (including a separate mouth set) and renders in real-time via Gaussian Splatting and Linear Blend Skinning using FLAME parameters. The method demonstrates competitive quantitative performance against photo-realistic baselines and yields high-quality stylized avatars in Arcane and Pixar styles, with favorable temporal stability and real-time rendering. Limitations include difficulty with side views from single-view training and potential improvements via additional training data and motion-capture integration.

Abstract

The introduction of 3D Gaussian blendshapes has enabled the real-time reconstruction of animatable head avatars from monocular video. Toonify, a StyleGAN-based method, has become widely used for facial image stylization. To extend Toonify for synthesizing diverse stylized 3D head avatars using Gaussian blendshapes, we propose an efficient two-stage framework, ToonifyGB. In Stage 1 (stylized video generation), we adopt an improved StyleGAN to generate the stylized video from the input video frames, which overcomes the limitation of cropping aligned faces at a fixed resolution as preprocessing for normal StyleGAN. This process provides a more stable stylized video, which enables Gaussian blendshapes to better capture the high-frequency details of the video frames, facilitating the synthesis of high-quality animations in the next stage. In Stage 2 (Gaussian blendshapes synthesis), our method learns a stylized neutral head model and a set of expression blendshapes from the generated stylized video. By combining the neutral head model with expression blendshapes, ToonifyGB can efficiently render stylized avatars with arbitrary expressions. We validate the effectiveness of ToonifyGB on benchmark datasets using two representative styles: Arcane and Pixar.

Figures (11)

• Figure 1: ToonifyGB: We propose an efficient two-stage framework that employs an improved StyleGAN to generate stylized head videos from input video frames and synthesize the corresponding 3D avatars using Gaussian blendshapes. Our method supports real-time synthesis of stylized avatar animations (with 50k Gaussians for the neutral model and 14k Gaussians for the mouth interior) in diverse styles such as Arcane and Pixar.
• Figure 2: Pipeline: Our ToonifyGB framework consists of two stages: Stage 1 involves the generation of stylized videos, and Stage 2 focuses on the synthesis of 3D stylized head avatars using Gaussian blendshapes.
• Figure 3: Visualization of stylized video generation results in "Arcane" and "Pixar" styles on the INSTA zielonka2023instant and NeRFBlendShape gao2022reconstructing datasets, covering both male and female subjects.
• Figure 4: Visualization of stylized video generation results: We present details of the real head from the input video, and the "Arcane" stylized head generated by our method. From left to right, the results for the video samples "bala" and "wojtek_1" are shown.
• Figure 5: Qualitative comparison of each stage: We present the input video head frames, the corresponding stylized videos, and 3D head avatars synthesized by our method.