Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets
Changjian Chen, Fei Lv, Yalong Guan, Pengcheng Wang, Shengjie Yu, Yifan Zhang, Zhuo Tang
TL;DR
Small-scale real-world image datasets constrain computer vision performance. This work introduces DataCrafter, a human-guided image generation framework that combines a multi-modal projection with content labels and a sample-level prompt refinement workflow to controllably expand datasets. It provides a visual analytics tool with metric monitoring, a multi-modal distribution view, and an interactive prompt refinement loop, underpinned by contrastive learning for robust cross-modal embeddings. Experimental results across classification and object detection tasks, plus expert feedback, demonstrate improved data quality and downstream performance, highlighting practical impact for real-world dataset expansion.
Abstract
The performance of computer vision models in certain real-world applications (e.g., rare wildlife observation) is limited by the small number of available images. Expanding datasets using pre-trained generative models is an effective way to address this limitation. However, since the automatic generation process is uncontrollable, the generated images are usually limited in diversity, and some of them are undesired. In this paper, we propose a human-guided image generation method for more controllable dataset expansion. We develop a multi-modal projection method with theoretical guarantees to facilitate the exploration of both the original and generated images. Based on the exploration, users refine the prompts and re-generate images for better performance. Since directly refining the prompts is challenging for novice users, we develop a sample-level prompt refinement method to make it easier. With this method, users only need to provide sample-level feedback (e.g., which samples are undesired) to obtain better prompts. The effectiveness of our method is demonstrated through the quantitative evaluation of the multi-modal projection method, improved model performance in the case study for both classification and object detection tasks, and positive feedback from the experts.