SynthSeg-Agents: Multi-Agent Synthetic Data Generation for Zero-Shot Weakly Supervised Semantic Segmentation

TL;DR

This work introduces SynthSeg-Agents, a zero-shot framework for weakly supervised semantic segmentation that entirely avoids real images by coupling an LLM-driven Self-Refine Prompt Agent with an image-generation pipeline and CLIP-based selection. A ViT-based classifier relabels the synthetic data to produce pseudo-labels, enabling downstream WSSS pipelines to train from scratch on fully synthetic data. Experiments on PASCAL VOC 2012 and MS COCO 2014 show competitive results with purely synthetic data, and ablations validate the contributions of prompt refinement and image relabeling. The approach demonstrates the viability of large-language-model–driven data generation for scalable, cost-efficient semantic segmentation in data-scarce or privacy-sensitive settings.

Abstract

Weakly Supervised Semantic Segmentation (WSSS) with image level labels aims to produce pixel level predictions without requiring dense annotations. While recent approaches have leveraged generative models to augment existing data, they remain dependent on real world training samples. In this paper, we introduce a novel direction, Zero Shot Weakly Supervised Semantic Segmentation (ZSWSSS), and propose SynthSeg Agents, a multi agent framework driven by Large Language Models (LLMs) to generate synthetic training data entirely without real images. SynthSeg Agents comprises two key modules, a Self Refine Prompt Agent and an Image Generation Agent. The Self Refine Prompt Agent autonomously crafts diverse and semantically rich image prompts via iterative refinement, memory mechanisms, and prompt space exploration, guided by CLIP based similarity and nearest neighbor diversity filtering. These prompts are then passed to the Image Generation Agent, which leverages Vision Language Models (VLMs) to synthesize candidate images. A frozen CLIP scoring model is employed to select high quality samples, and a ViT based classifier is further trained to relabel the entire synthetic dataset with improved semantic precision. Our framework produces high quality training data without any real image supervision. Experiments on PASCAL VOC 2012 and COCO 2014 show that SynthSeg Agents achieves competitive performance without using real training images. This highlights the potential of LLM driven agents in enabling cost efficient and scalable semantic segmentation.

Figures (6)

• Figure 1: Comparison of WSSS frameworks. Traditional methods rely on real images and CAMs, while our SynthSeg-Agents framework generates a synthetic dataset via LLM/VLM collaboration, requiring no real data.
• Figure 2: The overall pipeline of SynthSeg-Agents. Given a set of target class labels (e.g., cat, dog, ...), the Self-Refine Prompt Agent first generates image descriptions using a template-guided prompt mechanism. These prompts are iteratively refined via LLMs and filtered using CLIP-based semantic ANN to enhance relevance and diversity. The selected prompts are passed to the Image Agent, which uses a VLM to generate synthetic images. A frozen CLIP model is used to supervise a lightweight classifier for assigning pseudo-labels to the generated images. The final image–label pairs are used to train a WSSS model without using any real images.
• Figure 3: CLIP-based pseudo-labeling. Given a generated image and its prompt, we compute CLIP similarity between the prompt/image and all VOC classes. Classes with high scores in both spaces are selected as pseudo-labels—in this case, dog and sofa.
• Figure 4: The final training data consists of synthetic images generated by VLMs, followed by relabeling using our image classifier. For example, when prompting for the horse class, the VLMs may generate images that include multiple objects. To address this, we apply our image classifier to relabel each image with a refined to ensure that the final dataset is cleaner and more suitable for downstream training.
• Figure 5: Synthetic images generated by SynthSeg-Agents for PASCAL VOC 2012 and MS COCO 2014.(a) Samples for selected PASCAL VOC classes. (b) Samples for selected MS COCO classes. Each pair shows the class label (left) and the corresponding synthetic image (right), generated using prompts from our Self-Refine Prompt Agent.