Supervised makeup transfer with a curated dataset: Decoupling identity and makeup features for enhanced transformation

TL;DR

This work tackles makeup transfer by addressing data quality, identity–makeup disentanglement, and controllability using a diffusion-based approach. It introduces a three-part solution: (1) a train–generate–filter–retrain pipeline to assemble a high-quality paired makeup dataset, (2) a diffusion framework with explicit identity–makeup decoupling, and (3) a Mixed-Guided Attention module enabling text-driven, region-specific makeup edits. The method demonstrates improved fidelity, identity preservation, and editing flexibility compared with baselines on multiple benchmarks, supported by user studies and ablations. The combination of curated data and controllable diffusion editing offers practical, scalable capabilities for realistic, identity-consistent makeup transfer in real-world applications.

Abstract

Diffusion models have recently shown strong progress in generative tasks, offering a more stable alternative to GAN-based approaches for makeup transfer. Existing methods often suffer from limited datasets, poor disentanglement between identity and makeup features, and weak controllability. To address these issues, we make three contributions. First, we construct a curated high-quality dataset using a train-generate-filter-retrain strategy that combines synthetic, realistic, and filtered samples to improve diversity and fidelity. Second, we design a diffusion-based framework that disentangles identity and makeup features, ensuring facial structure and skin tone are preserved while applying accurate and diverse cosmetic styles. Third, we propose a text-guided mechanism that allows fine-grained and region-specific control, enabling users to modify eyes, lips, or face makeup with natural language prompts. Experiments on benchmarks and real-world scenarios demonstrate improvements in fidelity, identity preservation, and flexibility. Examples of our dataset can be found at: https://makeup-adapter.github.io.

Figures (6)

• Figure 1: Limitations of existing datasets and methods.
• Figure 2: Framework overview. Middle: training pipeline with disentanglement and reconstruction. Left: prompt-based training, where region-specific prompts (eyes, face, lips...) map to target images. Right: Mixed-Guided Attention (MGA) combining text, identity, and makeup features.
• Figure 3: Examples of images using preset makeup and fixing ledit++ generated image.
• Figure 4: Examples of dataset augmentation using the generate–filter strategy, where generated results are compared with LEDITS++ references and low-similarity samples are discarded.
• Figure 5: visualization of comparing to other methods.