BeautyBank: Encoding Facial Makeup in Latent Space

TL;DR

BeautyBank is proposed, a novel makeup encoder that disentangles pattern features of bare and makeup faces into a high-dimensional space, preserving essential details necessary for makeup reconstruction and broadening the scope of potential makeup research applications.

Abstract

The advancement of makeup transfer, editing, and image encoding has demonstrated their effectiveness and superior quality. However, existing makeup works primarily focus on low-dimensional features such as color distributions and patterns, limiting their versatillity across a wide range of makeup applications. Futhermore, existing high-dimensional latent encoding methods mainly target global features such as structure and style, and are less effective for tasks that require detailed attention to local color and pattern features of makeup. To overcome these limitations, we propose BeautyBank, a novel makeup encoder that disentangles pattern features of bare and makeup faces. Our method encodes makeup features into a high-dimensional space, preserving essential details necessary for makeup reconstruction and broadening the scope of potential makeup research applications. We also propose a Progressive Makeup Tuning (PMT) strategy, specifically designed to enhance the preservation of detailed makeup features while preventing the inclusion of irrelevant attributes. We further explore novel makeup applications, including facial image generation with makeup injection and makeup similarity measure. Extensive empirical experiments validate that our method offers superior task adaptability and holds significant potential for widespread application in various makeup-related fields. Furthermore, to address the lack of large-scale, high-quality paired makeup datasets in the field, we constructed the Bare-Makeup Synthesis Dataset (BMS), comprising 324,000 pairs of 512x512 pixel images of bare and makeup-enhanced faces.

Figures (20)

• Figure 1: Example applications of our makeup encoder (BeautyBank). We have successfully explored a variety of applications, including using (a) images with reference makeup to (b) generate facial images with makeup injection, (c) measure makeup similarity, and (d) transfer makeup, and (e) remove makeup. Additionally, BeautyBank can utilize two different facial identity references (Source Img 1 and 2) and two different makeup references (Ref Img 1 and 2) to (f) simultaneously interpolate identity and makeup. The images generated using the makeup code from BeautyBank show high-quality details such as makeup colors, patterns, and textures across various makeup applications.
• Figure 2: Typical issues in generated images using the baseline method. When DualStyleGAN dualstylegan is utilized for makeup transfer tasks, the generated images often exhibit inconsistencies in the facial identity compared to the source images. There is also a lack of detail in makeup attributes, such as local colors and patterns, and an entanglement with features that are not related to the makeup pattern.
• Figure 3: The workflow of latent code optimization. We enhance the encoding of identity information to optimize the bare-face code (see Section \ref{['sec:face code optimization']} for details). Subsequently, based on the encoded bare-face code, we use the specially designed objective function to enhance the encoding of makeup details and avoid encoding features unrelated to the makeup, achieving the final makeup encoding (see Section \ref{['sec:PMT']} for details).
• Figure 4: Example of bare-face encoding. Bare-face encoding results from (a) in the preliminary stage (in Section \ref{['sec:Preliminary_destylization']}) are shown in (b), while results from bare-face code optimization (in Section \ref{['sec:face code optimization']}) are shown in (c) and (d). Bare-face encoding progressively disentangles the makeup information contained in (a) while maintaining consistent identity features.
• Figure 5: Example of the Masks Utilized in BeautyBank. During Bare-face Code Optimization, the objective function employs mask (a) (in Section \ref{['sec:face code optimization']}). Stage 1 of Progressive Makeup Tuning utilizes masks (a), (b), and (c), while Stage 2 employs masks ranging from (a) to (f) (in Section \ref{['sec:PMT']}).