M$^3$Face: A Unified Multi-Modal Multilingual Framework for Human Face Generation and Editing

TL;DR

M3Face tackles the challenge of controllable, multilingual face generation and editing without requiring users to manually craft conditioning inputs. It unifies text-driven conditioning generation with a ControlNet-based generation path and an Imagic-based editing path, enabling both multi-modal and text-only workflows. The M3CelebA dataset supports multilingual captions and rich conditioning annotations, enhancing cross-language accessibility. Empirical results demonstrate state-of-the-art Qualitative and Quantitative performance in generation and editing, with zero-shot capabilities and detailed ablations. This work offers a practical, globally accessible framework for realistic face synthesis and manipulation, while acknowledging ethical considerations and platform limitations that guide responsible use.

Abstract

Human face generation and editing represent an essential task in the era of computer vision and the digital world. Recent studies have shown remarkable progress in multi-modal face generation and editing, for instance, using face segmentation to guide image generation. However, it may be challenging for some users to create these conditioning modalities manually. Thus, we introduce M3Face, a unified multi-modal multilingual framework for controllable face generation and editing. This framework enables users to utilize only text input to generate controlling modalities automatically, for instance, semantic segmentation or facial landmarks, and subsequently generate face images. We conduct extensive qualitative and quantitative experiments to showcase our frameworks face generation and editing capabilities. Additionally, we propose the M3CelebA Dataset, a large-scale multi-modal and multilingual face dataset containing high-quality images, semantic segmentations, facial landmarks, and different captions for each image in multiple languages. The code and the dataset will be released upon publication.

Figures (16)

• Figure 1: We introduce M3Face for controllable multi-modal multilingual face generation and editing. (a) Face Generation can be done with both multi-modal conditions or a single text input. We generate face images consistent with the input text and other given modalities. (b) Face Editing can also be done with text, mask, landmarks, or a combination of them.
• Figure 2: M3Face Framework. For (a) Face Generation, we first generate facial landmarks or semantic segmentation masks with a given text input with our Muse chang2023muse model. We then utilize our ControlNet zhang2023adding model to generate face images from the intermediate results. For (b) Face Editing, we utilize the Imagic kawar2023imagic method to manipulate images generated by our ControlNet. Face editing can be done with text, mask, landmarks, or a combination of them.
• Figure 3: Dataset Generation Pipeline. We first align, crop, and upscale the original CelebA images. We then generate the facial landmarks and semantic segmentation for each image. For generating the captions, we use the 40 CelebA attributes and utilize the GPT3.5 model for generation and the SeamlessM4T model for translation.
• Figure 4: M3CelebA Dataset.$512\times512$ images from the M3CelebA dataset as well as the generated facial landmarks and semantic segmentation. Three multilingual captions are available for each image.
• Figure 5: Face Generation Results. Our method generates realistic images based on the input prompt and the conditioning modality. We can generate faces consistent with semantic segmentation and facial landmarks. It also captures difficult attributes in the input prompt or the segmentation, such as glasses, hair color and style, and different face directions.