Portrait Video Editing Empowered by Multimodal Generative Priors

TL;DR

This work lifts the portrait video frames to a unified dynamic 3D Gaussian field, which ensures structural and temporal coherence across frames, and designs a novel Neural Gaussian Texture mechanism that not only enables sophisticated style editing but also achieves rendering speed over 100FPS.

Abstract

We introduce PortraitGen, a powerful portrait video editing method that achieves consistent and expressive stylization with multimodal prompts. Traditional portrait video editing methods often struggle with 3D and temporal consistency, and typically lack in rendering quality and efficiency. To address these issues, we lift the portrait video frames to a unified dynamic 3D Gaussian field, which ensures structural and temporal coherence across frames. Furthermore, we design a novel Neural Gaussian Texture mechanism that not only enables sophisticated style editing but also achieves rendering speed over 100FPS. Our approach incorporates multimodal inputs through knowledge distilled from large-scale 2D generative models. Our system also incorporates expression similarity guidance and a face-aware portrait editing module, effectively mitigating degradation issues associated with iterative dataset updates. Extensive experiments demonstrate the temporal consistency, editing efficiency, and superior rendering quality of our method. The broad applicability of the proposed approach is demonstrated through various applications, including text-driven editing, image-driven editing, and relighting, highlighting its great potential to advance the field of video editing. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/PortraitGen/

Figures (10)

• Figure 1: A totally 3D consistent model may not be an ideal solution for some styles. Many styles include intricate brush strokes and contour lines, which are actually not 3D consistent. Given the instruction 'Turn her into pixel style', our edited portrait could exhibit pixel contour lines, which is crucial for this kind of stylization.
• Figure 2: We first track the SMPL-X coefficients of the given monocular video, and then use a Neural Gaussian Texture mechanism to get a 3D Gaussian feature field. These neural Gaussians are further splatted to render portrait images. An iterative dataset update strategy is applied for portrait editing, and a Multimodal Face Aware Editing module is proposed to enhance expression quality and preserve personalized facial structures.
• Figure 3: The Neural Renderer could effectively combine the information of splatted Gaussians and further improve the representation ability of 3D Gaussian portrait representation. With our Neural Gaussian Texture mechanism, the edited portrait follows prompts better and exhibit higher quality. (given instruction: Turn him into Lego style)
• Figure 4: We alternate between editing the dataset of video frames and updating the underlying 3D portrait. The portrait model will gradually converge to the target prompt, achieving both 3D and temporal consistency.
• Figure 5: Qualitative comparisons on text driven portrait editing.