PEGASUS: Personalized Generative 3D Avatars with Composable Attributes

TL;DR

PEGASUS addresses the challenge of creating personalized, editable 3D face avatars from monocular video. It introduces a two-stage pipeline: (1) synthetic DB generation by part swapping from diverse identities, and (2) learning a personalized generative avatar with disentangled latent codes that control subparts while preserving identity. A zero-shot transfer variant enables efficient generation by reusing a previously trained model. Across extensive experiments, PEGASUS demonstrates strong identity preservation and realistic rendering, with effective disentanglement across hair, nose, and other attributes, though it acknowledges artifacts and non-physical synthesis as limitations with potential for future improvement.

Abstract

We present PEGASUS, a method for constructing a personalized generative 3D face avatar from monocular video sources. Our generative 3D avatar enables disentangled controls to selectively alter the facial attributes (e.g., hair or nose) while preserving the identity. Our approach consists of two stages: synthetic database generation and constructing a personalized generative avatar. We generate a synthetic video collection of the target identity with varying facial attributes, where the videos are synthesized by borrowing the attributes from monocular videos of diverse identities. Then, we build a person-specific generative 3D avatar that can modify its attributes continuously while preserving its identity. Through extensive experiments, we demonstrate that our method of generating a synthetic database and creating a 3D generative avatar is the most effective in preserving identity while achieving high realism. Subsequently, we introduce a zero-shot approach to achieve the same goal of generative modeling more efficiently by leveraging a previously constructed personalized generative model.

Figures (22)

• Figure 1: PEGASUS. Our method builds a personalized generative 3D face avatar from monocular video sources.
• Figure 2: Method Overview. Our approach consists of two main components: synthetic database (DB) generation and a personalized generative 3D avatar model. Initially, we build a synthetic DB via face part swapping from the attribute DB videos. For the generation of the synthetic DB, we propose the method through post-processing and attribute alignment leveraging FLAME parameters. Subsequently, we train our model utilizing the synthetic DB that contains the same target identity with varying attributes. Our model infers the 3D point locations in the deformed space $\mathbf{x}^d$, normal $\mathbf{n}^d$, shading $\mathbf{s}^d$, point segment cues $\chi^d$, and the albedo $\mathbf{a}^d$ for each queried canonical point $\mathbf{x}^{gc}$, conditioned by the latent code $\mathbf{z}$.
• Figure 3: DB Avatar. We create deformable avatar models from the attribute DB videos, which are monocular RGB inputs. We show some examples of our collection of avatars from attribute DB videos with the same FLAME parameters.
• Figure 4: Part-Swapped Videos of the Target Individual. Some examples of the synthetic DB created through part-swapping. Our synthetic DB includes a variety of hair, hats, eyes, noses, mouths, and eyebrows.
• Figure 5: Zero Shot Transfer. PEGASUS generates high-quality and natural appearances through zero-shot transfer.