ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentation

TL;DR

Addressing the data bottleneck in scribble-supervised semantic segmentation, the paper introduces ScribbleGen, a diffusion-based data augmentation framework conditioned on scribbles. It uses a ControlNet diffusion model with classifier-free guidance and an encode ratio to generate a spectrum of synthetic images with controllable realism and diversity, and combines them with real data via fixed or adaptive schedules using a guidance scale $w$ and encode ratio $\lambda$. The approach yields state-of-the-art results on Pascal VOC for scribble-supervised methods (RLoss and AGMM), substantially narrowing the gap to fully-supervised segmentation and sometimes surpassing it in low-data regimes. This work demonstrates a practical pathway to data-efficient segmentation and opens avenues for open-vocabulary conditioning and semi-supervised extensions.

Abstract

Recent advances in generative models, such as diffusion models, have made generating high-quality synthetic images widely accessible. Prior works have shown that training on synthetic images improves many perception tasks, such as image classification, object detection, and semantic segmentation. We are the first to explore generative data augmentations for scribble-supervised semantic segmentation. We propose ScribbleGen, a generative data augmentation method that leverages a ControlNet diffusion model conditioned on semantic scribbles to produce high-quality training data. However, naive implementations of generative data augmentations may inadvertently harm the performance of the downstream segmentor rather than improve it. We leverage classifier-free diffusion guidance to enforce class consistency and introduce encode ratios to trade off data diversity for data realism. Using the guidance scale and encode ratio, we can generate a spectrum of high-quality training images. We propose multiple augmentation schemes and find that these schemes significantly impact model performance, especially in the low-data regime. Our framework further reduces the gap between the performance of scribble-supervised segmentation and that of fully-supervised segmentation. We also show that our framework significantly improves segmentation performance on small datasets, even surpassing fully-supervised segmentation. The code is available at https://github.com/mengtang-lab/scribblegen.

Figures (12)

• Figure 2: Given a limited number of real scribbles, we pretrain a ControlNet-based diffusion model for high-fidelity synthesis of images conditioned on scribbles. We can control the image synthesis with the encode ratio $\lambda$ and the guidance scale $w$. These image-scribble pairs can then be smoothly integrated into the training of scribble-based semantic segmentation.
• Figure 3: Left: Our sampled synthetic images conditioned on the ground-truth scribble. By sampling using different guidance scales and encode ratios we are able to generate a whole spectrum of realistic synthetic training images. Right: The ground-truth real image and corresponding scribble label.
• Figure 4: Synthetic images sampled from diffusion models with different numbers of training images and encode ratios $\lambda$.
• Figure 5: Left: The FID of our full training dataset when generated with different classifier-free guidance scales. Results are reported for ControlNet trained all 10582 images. Middle: The FID of our training dataset when generated with different encode ratios. Results are reported for four ControlNet models trained on a different number of images. Right: The mIoU of a downstream segmentation model when trained on datasets of varying encode ratios. Note $\lambda=0.0$ corresponds to training on real images only. Results are reported for training on both naive data augmentation and only on synthetic images. In both cases, we use all 10582 images for training.
• Figure 6: Qualitative results on PASCAL dataset. Our generative data augmentation method improves scribble-supervised semantic segmentation methods such as AGMM AGMM.