A Training-free Synthetic Data Selection Method for Semantic Segmentation

TL;DR

This work tackles the problem of leveraging synthetic data for semantic segmentation by proposing a training-free Synthetic Data Selection (SDS) framework that uses CLIP-based filters. It introduces two modules: Perturbation-based CLIP Similarity (PCS) to assess image reliability via differences in text–image similarity before and after patch mixing, and a Class-balance Annotation Similarity Filter (ASF) to prune low-quality annotations by comparing synthetic annotations to CLIP-derived references using mIoU. The approach enables significant data reduction (roughly half the initial synthetic data) while achieving higher segmentation performance on VOC 2012 and COCO 2017 across multiple backbones and segmentation models. The results demonstrate that selective data curation with PCS and ASF yields practical gains, reducing reliance on large, potentially noisy synthetic datasets for semantic segmentation.

Abstract

Training semantic segmenter with synthetic data has been attracting great attention due to its easy accessibility and huge quantities. Most previous methods focused on producing large-scale synthetic image-annotation samples and then training the segmenter with all of them. However, such a solution remains a main challenge in that the poor-quality samples are unavoidable, and using them to train the model will damage the training process. In this paper, we propose a training-free Synthetic Data Selection (SDS) strategy with CLIP to select high-quality samples for building a reliable synthetic dataset. Specifically, given massive synthetic image-annotation pairs, we first design a Perturbation-based CLIP Similarity (PCS) to measure the reliability of synthetic image, thus removing samples with low-quality images. Then we propose a class-balance Annotation Similarity Filter (ASF) by comparing the synthetic annotation with the response of CLIP to remove the samples related to low-quality annotations. The experimental results show that using our method significantly reduces the data size by half, while the trained segmenter achieves higher performance. The code is released at https://github.com/tanghao2000/SDS.

Figures (6)

• Figure 1: Comparison of synthetic data training methods. (a) Previous methods use all the synthetic data to train a segmenter. (b) Our training-free synthetic data selection method (SDS) selects higher-quality samples to train the segmented, where we select images with significant text-to-image similarity differences before and after mixed image patches.
• Figure 2: Overview of our Synthetic Data Selection (SDS) framework. (1) The initial synthetic dataset consists of 40k image-annotation pairs generated by a generative model. (2) Our SDS includes two modules: PCS and ASF. In the PCS module, the text caption and original image are encoded to feature represents $F_t$ and $F_x$. For patch-mixed image, the multi-level patch-mixed strategy are designed to obtain $F_{\hat{x_j}}$. We calculate the text-to-image similarity and select image samples with high similarity and high PCS scores. In the ASF module, we design the rule (a) and (b) to calculate the mIoU between synthetic annotations and the response of CLIP. The mIoU scores are sorted and we select the Top $n$ annotation samples. (3) The selected dataset remaining 26k images-annotation pairs are used to train a segmenter.
• Figure 3: Visualization of selected images. High-quality images are with green box and low-quality with red box.
• Figure 4: Visualization of low-quality annotation examples. (a) Synthetic Images. (b) Synthetic Annotations. (c) Reference Annotations from CLIP.
• Figure 5: Segmentation results on the PASCAL VOC 2012 validation set. (a) Images. (b) Ground-truth annotation. (c) Predicted annotation from Dataset Diffusion. (d) Our predicted annotation.