Protégé: Learn and Generate Basic Makeup Styles with Generative Adversarial Networks (GANs)

TL;DR

Protégé introduces a GAN-based makeup-inpainting framework that learns basic makeup styles from a curated dataset and generates individualized makeup while preserving facial identity. The method uses a binary ROI mask and a makeup-free face reconstruction to apply makeup through a StyleGAN2-based generator, ensuring style alignment with the target dataset and identity preservation. Training employs adversarial and perceptual losses with explicit weighting to balance realism and identity consistency, and is validated on the BeautyFace dataset with favorable FID and ArcFace metrics and positive user reception. This approach advances digital makeup by delivering intuitive, diverse, and customizable makeup generation beyond traditional manual, rule-based, or transfer-based systems, with potential for industry-specific digital makeup artists and applications.

Abstract

Makeup is no longer confined to physical application; people now use mobile apps to digitally apply makeup to their photos, which they then share on social media. However, while this shift has made makeup more accessible, designing diverse makeup styles tailored to individual faces remains a challenge. This challenge currently must still be done manually by humans. Existing systems, such as makeup recommendation engines and makeup transfer techniques, offer limitations in creating innovative makeups for different individuals "intuitively" -- significant user effort and knowledge needed and limited makeup options available in app. Our motivation is to address this challenge by proposing Protégé, a new makeup application, leveraging recent generative model -- GANs to learn and automatically generate makeup styles. This is a task that existing makeup applications (i.e., makeup recommendation systems using expert system and makeup transfer methods) are unable to perform. Extensive experiments has been conducted to demonstrate the capability of Protégé in learning and creating diverse makeups, providing a convenient and intuitive way, marking a significant leap in digital makeup technology!

Figures (5)

• Figure 1: Overview of Protégé. (Top left) Protégé learns basic makeup styles from a curated makeup dataset. (Top right) After training, the model applies its learned styles and knowledge to generate and apply makeup on individual faces. (Bottom) Additional examples of makeup generated by Protégé.
• Figure 2: Methodology Overview of Protégé Makeup Learning and Generation Process. This figure illustrates the key stages of the process in Protégé. First, a binary Region of Interest (ROI) mask is created to identify facial areas for makeup. Protégé then comprehends the makeup style by generating a makeup-free face and blending it with the original image. It synthesizes a diverse range of makeup styles via a GAN-based generator, ensuring high-quality, personalized makeup while preserving facial identity. Finally, makeup is seamlessly integrated with the untouched non-facial areas to produce the final output image. Loss functions are employed to ensure stylistic accuracy and maintain the authenticity of the makeup. This framework enables Protégé to effectively learn and generate innovative makeup styles tailored to individual faces.
• Figure 3: Visual Comparison between generated makeup by Protégé and makeup done by human makeup artist.
• Figure 4: Close-up views of Protégé's makeups on makeup-free faces such as eyebrows, eyeliner, eyeshadow, blush, and lipstick, highlighting its finesse and natural-looking outcomes.
• Figure 5: Visual Comparisons in Face Identity Preservation: Which of these faces belongs to the same person as the one shown in the makeup-free image?