DreamMakeup: Face Makeup Customization using Latent Diffusion Models

TL;DR

DreamMakeup tackles the problem of customizable realistic face makeup without model fine tuning. It leverages a latent diffusion prior and early stopped DDIM inversion to preserve identity while enabling local pixel space edits and text guided global harmonization via cross attention. The approach supports RGB color, reference image, and textual conditioning and demonstrates compatibility with LLMs and LoRAs, achieving faster inference on common GPUs. Empirical results show superior color fidelity and identity preservation against GAN based and diffusion based baselines, with ablations validating the influence of t star, interpolation lambda and text guidance. The method promises practical deployment for personalized virtual makeup and extensible integration with other AI systems.

Abstract

The exponential growth of the global makeup market has paralleled advancements in virtual makeup simulation technology. Despite the progress led by GANs, their application still encounters significant challenges, including training instability and limited customization capabilities. Addressing these challenges, we introduce DreamMakup - a novel training-free Diffusion model based Makeup Customization method, leveraging the inherent advantages of diffusion models for superior controllability and precise real-image editing. DreamMakeup employs early-stopped DDIM inversion to preserve the facial structure and identity while enabling extensive customization through various conditioning inputs such as reference images, specific RGB colors, and textual descriptions. Our model demonstrates notable improvements over existing GAN-based and recent diffusion-based frameworks - improved customization, color-matching capabilities, identity preservation and compatibility with textual descriptions or LLMs with affordable computational costs.

Figures (19)

• Figure 1: We present DreamMakeup, a training-free diffusion framework that generates high-fidelity makeup results by integrating diverse user inputs such as RGB colors, reference images, and text prompts. Our method produces high-quality, customized makeup while preserving facial identity, without requiring any fine-tuning. Please zoom in for detailed inspection.
• Figure 2: Overview of DreamMakeup pipeline. The key principle of our framework is to apply fine-grained guidance in high-dimensional pixel-domain during reverse sampling. After local makeup customization in pixel space, text prompts are leveraged to harmonize such local variations with a consistent global style in latent cross-attention space.
• Figure 3: (a) Direct pixel-space customization results in color inconsistencies, while SDEdit (strength=0.2) degrades the subject's identity. In contrast, our method applies the makeup faithfully while preserving identity. (b) The impact of the interpolation domain during reverse sampling. Latent-space interpolation (b-1) effectively preserves fine-grained facial details, whereas pixel-space interpolation (b-2) introduces significant visual artifacts. Further details are provided in \ref{['sec: EBCQ']}.
• Figure 4: (a) Color-based makeup transformation. Mean RGB values within the masked area are adjusted with a scale $\alpha$ to match the target RGB values. Output image is generated by reverse sampling from $\hat{{\boldsymbol x}}_{new}$. (b) Eyeshadow mask is reproduced from eye mask manipulation.
• Figure 5: The virtual skin, lip, eye shadow makeup, and their combination by DreamMakeup (SD 1.5).