MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing

TL;DR

MaTe3D introduces a mask-guided and text-based 3D-aware portrait editing framework built around an SDF-based generator that learns global and local representations via dedicated $L_{sdf}$ and $L_{\sigma}$ losses, coupled with a CDGT distillation strategy to stabilize texture and geometry during edits. The approach combines dual tri-planes and neural rendering to produce coherent 3D geometry and semantic masks, while an inference-optimized editing stage fuses a frozen and a learnable generator with Condition Updating and gradient blending to maintain multi-view consistency under mask and text prompts. A new CatMask-HQ dataset enables evaluation beyond human faces, and experiments on FFHQ and CatMask-HQ show competitive image quality, strong view consistency, and effective mask-text editing with ablations confirming the contributions of SDF representation, density consistency, and CDGT. The work advances practical 3D-aware portrait editing with joint mask/text control, offering a scalable path toward real-world applications in portrait manipulation, out-of-domain editing, and face swapping, albeit with some computational cost and occasional geometry trade-offs. Overall, MaTe3D provides a robust framework for high-quality, stable 3D portrait editing that leverages SDF geometry and diffusion priors to fuse spatial precision with semantic edits.

Abstract

3D-aware portrait editing has a wide range of applications in multiple fields. However, current approaches are limited due that they can only perform mask-guided or text-based editing. Even by fusing the two procedures into a model, the editing quality and stability cannot be ensured. To address this limitation, we propose \textbf{MaTe3D}: mask-guided text-based 3D-aware portrait editing. In this framework, first, we introduce a new SDF-based 3D generator which learns local and global representations with proposed SDF and density consistency losses. This enhances masked-based editing in local areas; second, we present a novel distillation strategy: Conditional Distillation on Geometry and Texture (CDGT). Compared to exiting distillation strategies, it mitigates visual ambiguity and avoids mismatch between texture and geometry, thereby producing stable texture and convincing geometry while editing. Additionally, we create the CatMask-HQ dataset, a large-scale high-resolution cat face annotation for exploration of model generalization and expansion. We perform expensive experiments on both the FFHQ and CatMask-HQ datasets to demonstrate the editing quality and stability of the proposed method. Our method faithfully generates a 3D-aware edited face image based on a modified mask and a text prompt. Our code and models will be publicly released.

Figures (13)

• Figure 1: Framework of MaTe3D. MaTe3D Generator (a) consists of Tri-planes Generation and Neural Rendering. Tri-planes Generation constructs 3D volumes of texture and shape in tri-plane representations. Neural Rendering renders 3D-aware image and mask from learned SDF and color. In Inference-optimized editing phase (b), we utilize both a frozen generator ($G_{frz}$) and a learnable generator ($G_{opt}$), which were initialized by the generator in (a). In addition, we propose Condition Distillation on Geometry and Texture (CDGT) to achieve mask-guided and text-based editing while maintaining consistent texture across multiple views and producing reasonable geometry.
• Figure 2: Dataset qualitative comparisons with an existing dataset. CatMask-HQ has superior annotation quality.
• Figure 3: Qualitative comparison in image quality. MaTe3D achieves comparable image quality and better view consistency across face and cat.
• Figure 4: Qualitative comparison with the state-of-the-art methods on decompositional geometries. MaTe3D has more consistent and well-decompositional meshes than FENeRF (staircase artifacts), IDE-3D (inaccurate density distribution along rays) and Ours(w/o SDF, inaccurate eye modeling). We merge some classes for demonstration purposes.
• Figure 5: Text-based Editing. Our method can edit images using text while preserving the mask stability, unlike other methods that fail to maintain an unchanged mask or synthesize unimpressive texture. Here, 'SC' represents 'StyleCLIP'.