Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language

TL;DR

ACP presents Auto Cherry-Picker, a language-driven pipeline that generates high-quality cross-modality training data by sampling object combinations with data priors, producing detailed scene graphs and descriptions via LLMs, and synthesizing images with controllable diffusion. It introduces CLIS, a comprehensive quality filter with CLIS-L for layouts and CLIS-I for images, to align synthetic data with real data distributions and improve downstream performance, especially in long-tail and open-vocabulary settings. Empirical results on COCO, LVIS, MME, and GQA show significant gains and a positive correlation between CLIS scores and downstream improvements, supporting the utility of generation metrics for data quality assessment. The work demonstrates scalable, annotation-light data augmentation that enhances perception and multi-modal reasoning tasks, with potential for further efficiency and metric refinement as diffusion and LLM capabilities advance.

Abstract

Diffusion models can generate realistic and diverse images, potentially facilitating data availability for data-intensive perception tasks. However, leveraging these models to boost performance on downstream tasks with synthetic data poses several challenges, including aligning with real data distribution, scaling synthetic sample volumes, and ensuring their quality. To bridge these gaps, we present \textbf{A}uto \textbf{C}herry-\textbf{P}icker (ACP), a novel framework that generates high-quality cross-modality training samples at scale to augment perception and multi-modal training. ACP first uses LLMs to sample descriptions and layouts based on object combinations from real data priors, eliminating the need for ground truth image captions or annotations. Next, we use an off-the-shelf controllable diffusion model to generate multiple images. Then, the generated data are refined using a comprehensively designed metric, Composite Layout and Image Score (CLIS), to ensure quality. Our customized synthetic high-quality samples boost performance in various scenarios, especially in addressing challenges associated with long-tailed distribution and imbalanced datasets. Experiment results on downstream tasks demonstrate that ACP can significantly improve the performance of existing models. In addition, we find a positive correlation between CLIS and performance gains in downstream tasks. This finding shows the potential for evaluation metrics as the role for various visual perception and MLLM tasks.

Figures (15)

• Figure 1: Comparison with previous methods. (a) LMD lian2024llmgrounded generates samples conditioned on detailed image descriptions by leveraging LLMs as the layout generator and diffusion-based models as the image generator. Some methods he2022synthetic use CLIP filtering to future refine these samples. (b) ACP synthesizes training samples conditioned solely on object combinations in natural language and automatically cherry-picks high-quality ones by evaluating both layouts and images. High-quality training samples are more effective for downstream tasks.
• Figure 2: Illustration of Auto Cherry-Picker pipeline. It contains a (a) raw data generator and a (b) data filter using CLIS. Conditioned on input object combination sampled from data priors, Scene Graph Generator generates detailed attributes, relations, captions, and corresponding layouts. Subsequently, the Image Generator produces images based on the scene graph. These raw layouts and images are refined through filters using CLIS-L and CLIS-I, respectively, to produce high-quality training data.
• Figure 3: Comparison of generation results based on the same input object combinations and synthetic descriptions with and without CLIS. More generation results can be found in Appx. \ref{['app:vis']}.
• Figure 4: Consistent with human judgement. See details in Appx. \ref{['appx:consistency']}.
• Figure 5: Correlation between CLIS and performance gains on downstream tasks. (a,b): Synthetic data with different ranges of CLIS-I on long-tailed instance segmentation and open-vocabulary detection scenarios of LVIS benchmarks using Mask R-CNN and Grounding-DINO as baseline, respectively. (c,d): Synthetic data with different ranges of CLIS-L on multi-modal perception and reasoning MME and GQA benchmarks based on LLaVA-v1.5.