Toward Tiny and High-quality Facial Makeup with Data Amplify Learning

TL;DR

The core idea of DAL lies in employing a Diffusion-based Data Amplifier to "amplify"limited images for the model training, thereby enabling accurate pixel-to-pixel supervision with merely a handful of annotations.

Abstract

Contemporary makeup approaches primarily hinge on unpaired learning paradigms, yet they grapple with the challenges of inaccurate supervision (e.g., face misalignment) and sophisticated facial prompts (including face parsing, and landmark detection). These challenges prohibit low-cost deployment of facial makeup models, especially on mobile devices. To solve above problems, we propose a brand-new learning paradigm, termed "Data Amplify Learning (DAL)," alongside a compact makeup model named "TinyBeauty." The core idea of DAL lies in employing a Diffusion-based Data Amplifier (DDA) to "amplify" limited images for the model training, thereby enabling accurate pixel-to-pixel supervision with merely a handful of annotations. Two pivotal innovations in DDA facilitate the above training approach: (1) A Residual Diffusion Model (RDM) is designed to generate high-fidelity detail and circumvent the detail vanishing problem in the vanilla diffusion models; (2) A Fine-Grained Makeup Module (FGMM) is proposed to achieve precise makeup control and combination while retaining face identity. Coupled with DAL, TinyBeauty necessitates merely 80K parameters to achieve a state-of-the-art performance without intricate face prompts. Meanwhile, TinyBeauty achieves a remarkable inference speed of up to 460 fps on the iPhone 13. Extensive experiments show that DAL can produce highly competitive makeup models using only 5 image pairs.

Figures (11)

• Figure 1: Our Diffusion-based Data Amplifier is capable of learning from just several images to generate high-quality makeup visuals in diverse styles, offering flexible control such as makeup shade control and customized local editing.
• Figure 2: Left: Data evolution in facial makeup: unpaired data$\rightarrow$pseudo-paired data$\rightarrow$paired data (generated by data amplifier). Right: Comparison between TinyBeauty trained on paired data and EleGANt ELEGANT trained on pseudo-paired data.
• Figure 3: Overview of the Data Amplify Learning framework. The Data Amplify Learning process contains two components: (1)A data amplifier which utilizes a pretrained diffusion model to amplify a small set of seed data into a larger synthesized dataset. (2) A lightweight model which is trained on the amplified data to accurately learn the makeup styles while retaining identity features of the original images. $^*$The latency is the inference time on an iPhone 13 device.
• Figure 4: Overview of the Diffusion-based Data Amplifier (DDA). Our DDA leverages a Residual Diffusion Model for high-fidelity texture preservation, minimizing distortion and avoiding unnatural mask-like appearances. It also employs a Fine-Grained Makeup Module including Identity Preservation Block (IPB) to maintain the original facial features, Style Preservation Block (SPB) to guarantee consistent makeup style application, and facial masks to specify makeup region.
• Figure 5: Visual Comparison of Facial Makeup Using DDA. Comparative results highlight the superior performance of DDA in maintaining facial integrity and style consistency when compared with alternative methods.