Image Synthesis under Limited Data: A Survey and Taxonomy

TL;DR

This survey addresses image synthesis under limited data by introducing a unified taxonomy that splits the problem into four tasks: data-efficient generative models, few-shot generative adaptation, few-shot image generation, and one-shot image synthesis. It analyzes core methodologies across augmentation, regularization, architecture design, and transfer-based strategies, and evaluates them on standard benchmarks using metrics like FID and LPIPS. The paper highlights the strengths and limitations of each approach, discusses practical applications such as editing and augmentation, and outlines future directions including controllability and better evaluation. By providing up-to-date benchmarks, comparisons, and a curated repository, it aims to guide researchers toward effective data-efficient generation in real-world settings.

Abstract

Deep generative models, which target reproducing the given data distribution to produce novel samples, have made unprecedented advancements in recent years. Their technical breakthroughs have enabled unparalleled quality in the synthesis of visual content. However, one critical prerequisite for their tremendous success is the availability of a sufficient number of training samples, which requires massive computation resources. When trained on limited data, generative models tend to suffer from severe performance deterioration due to overfitting and memorization. Accordingly, researchers have devoted considerable attention to develop novel models that are capable of generating plausible and diverse images from limited training data recently. Despite numerous efforts to enhance training stability and synthesis quality in the limited data scenarios, there is a lack of a systematic survey that provides 1) a clear problem definition, critical challenges, and taxonomy of various tasks; 2) an in-depth analysis on the pros, cons, and remain limitations of existing literature; as well as 3) a thorough discussion on the potential applications and future directions in the field of image synthesis under limited data. In order to fill this gap and provide a informative introduction to researchers who are new to this topic, this survey offers a comprehensive review and a novel taxonomy on the development of image synthesis under limited data. In particular, it covers the problem definition, requirements, main solutions, popular benchmarks, and remain challenges in a comprehensive and all-around manner.

Figures (7)

• Figure 1: Various problem settings of image synthesis models under limited data. In particular, (a) represents the data-efficient GAN training pipeline that learns to capture the observed distribution from scratch with limited data; (b) denotes the pipeline of few-shot generative adaptation, which transfers prior knowledge from pre-trained large-scale source generative models to target domains with very few images, e.g., 10-shot images; (c) shows the learning scheme of few-shot image generation, the model is expected to produce novel samples given several input conditional images; (d) presents the training process of one-shot image generation, the model is trained solely on one single image in a coarse-to-fine manner to capture the underlying internal distribution of the reference image. It is imperative to note that the sub-figures presented herein serve solely as illustrative aids to convey the problem settings of diverse image synthesis tasks. As such, it should be understood that not all approaches utilized in these tasks are consistent with the pipeline depicted in the diagrams.
• Figure 2: An overview of various deep generative models.
• Figure 3: The general concepts of few-shot learning and transfer learning.
• Figure 4: Qualitative comparisons of different one-shot generative models. Images quoted from kulikov2023sinddm.
• Figure 5: Generative models under limited data have enabled downstream applications in various image editing tasks, such as paint-to-image, image harmonization, super-resolution, and animation. Images quoted from singan.