Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing AI Face Realism: Cost-Efficient Quality Improvement in Distilled Diffusion Models with a Fully Synthetic Dataset

Jakub Wasala, Bartlomiej Wrzalski, Kornelia Noculak, Yuliia Tarasenko, Oliwer Krupa, Jan Kocon, Grzegorz Chodak

TL;DR

This work tackles the high computational cost of diffusion-based portrait generation by pairing a lightweight distilled backbone (FLUX.1-schnell) with a dedicated image-to-image refinement head trained on a fully synthetic paired dataset to reach the fidelity of a heavier baseline (FLUX.1-dev). It introduces two I2I training paradigms—supervised paired U-Net and unsupervised CycleGAN variants (including ESA-CycleGAN)—and a novel fully synthetic dataset generation approach that yields 280,000 paired portraits. The results show that the distilled-plus-I2I pipeline can achieve photorealistic portraits with substantial runtime savings (up to $82\%$) while maintaining high perceptual quality, as measured by $FID_{diff}$ and related metrics. The study provides a model-agnostic, cost-efficient pathway to deploy high-quality diffusion outputs in resource-constrained settings and suggests broad applicability to other domains with careful dataset construction and evaluation.

Abstract

This study presents a novel approach to enhance the cost-to-quality ratio of image generation with diffusion models. We hypothesize that differences between distilled (e.g. FLUX.1-schnell) and baseline (e.g. FLUX.1-dev) models are consistent and, therefore, learnable within a specialized domain, like portrait generation. We generate a synthetic paired dataset and train a fast image-to-image translation head. Using two sets of low- and high-quality synthetic images, our model is trained to refine the output of a distilled generator (e.g., FLUX.1-schnell) to a level comparable to a baseline model like FLUX.1-dev, which is more computationally intensive. Our results show that the pipeline, which combines a distilled version of a large generative model with our enhancement layer, delivers similar photorealistic portraits to the baseline version with up to an 82% decrease in computational cost compared to FLUX.1-dev. This study demonstrates the potential for improving the efficiency of AI solutions involving large-scale image generation.

Enhancing AI Face Realism: Cost-Efficient Quality Improvement in Distilled Diffusion Models with a Fully Synthetic Dataset

TL;DR

This work tackles the high computational cost of diffusion-based portrait generation by pairing a lightweight distilled backbone (FLUX.1-schnell) with a dedicated image-to-image refinement head trained on a fully synthetic paired dataset to reach the fidelity of a heavier baseline (FLUX.1-dev). It introduces two I2I training paradigms—supervised paired U-Net and unsupervised CycleGAN variants (including ESA-CycleGAN)—and a novel fully synthetic dataset generation approach that yields 280,000 paired portraits. The results show that the distilled-plus-I2I pipeline can achieve photorealistic portraits with substantial runtime savings (up to ) while maintaining high perceptual quality, as measured by and related metrics. The study provides a model-agnostic, cost-efficient pathway to deploy high-quality diffusion outputs in resource-constrained settings and suggests broad applicability to other domains with careful dataset construction and evaluation.

Abstract

This study presents a novel approach to enhance the cost-to-quality ratio of image generation with diffusion models. We hypothesize that differences between distilled (e.g. FLUX.1-schnell) and baseline (e.g. FLUX.1-dev) models are consistent and, therefore, learnable within a specialized domain, like portrait generation. We generate a synthetic paired dataset and train a fast image-to-image translation head. Using two sets of low- and high-quality synthetic images, our model is trained to refine the output of a distilled generator (e.g., FLUX.1-schnell) to a level comparable to a baseline model like FLUX.1-dev, which is more computationally intensive. Our results show that the pipeline, which combines a distilled version of a large generative model with our enhancement layer, delivers similar photorealistic portraits to the baseline version with up to an 82% decrease in computational cost compared to FLUX.1-dev. This study demonstrates the potential for improving the efficiency of AI solutions involving large-scale image generation.
Paper Structure (20 sections, 3 equations, 7 figures, 4 tables)

This paper contains 20 sections, 3 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion-Based Image-to-Image Translation
  4. FLUX: A Computationally Efficient Generative Model
  5. Fine-Tuning Techniques
  6. Gaps and Contributions
  7. Dataset
  8. Preparation of the dataset
  9. Analysis of Dataset Diversity
  10. Architecture
  11. Supervised Pairwise Approach
  12. Unsupervised Non-Pairwise Approach
  13. Experiments
  14. Supervised Pairwise Training
  15. Unsupervised Non-pairwise Training
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Left sides: example input images from FLUX.1-schnell. Right sides: outputs from our model based on ESA-CycleGAN. Our model adds skin details, enhances hair, and improves the reflection of eye pupils, contributing to the overall realistic feel of a portrait.
  • Figure 2: Example pair of FLUX.1-schnell and FLUX.1-dev images from our dataset. We observe that the dev variant generates portraits featuring more details of skin, hair, and eyes, leading to a more photorealistic look.
  • Figure 3: t-SNE visualization of face embeddings. Representations were acquired using the ArcFace model deng2019arcface.
  • Figure 4: Pipeline of our solution, including the integration of FLUX.1-schnell and our image-to-image model.
  • Figure 5: Single block of our image-to-image U-Net model.
  • ...and 2 more figures