A Framework for Low-Effort Training Data Generation for Urban Semantic Segmentation

TL;DR

This work tackles the gap between synthetic urban data and real-world imagery by introducing a diffusion-model framework that adapts to a target domain using unlabelled images and imperfect pseudo-labels. It employs a two-stage fine-tuning to separate appearance alignment from semantic conditioning, augmented with regularisation (erosion, depth cues, CFG) and an automated, diverse data-generation pipeline guided by Mean Class-wise Object Consistency. Across five synthetic sources translated to Cityscapes and ACDC, the method consistently outperforms state-of-the-art I2I approaches, delivering up to +8.0 percentage points in mIoU and enabling high-quality training data from low-effort synthetic scenes. The approach emphasizes transparency, reusability, and rapid creation of rare or safety-critical scenarios, suggesting a collaborative paradigm between 3D asset creation and generative modelling for scalable urban scene understanding.

Abstract

Synthetic datasets are widely used for training urban scene recognition models, but even highly realistic renderings show a noticeable gap to real imagery. This gap is particularly pronounced when adapting to a specific target domain, such as Cityscapes, where differences in architecture, vegetation, object appearance, and camera characteristics limit downstream performance. Closing this gap with more detailed 3D modelling would require expensive asset and scene design, defeating the purpose of low-cost labelled data. To address this, we present a new framework that adapts an off-the-shelf diffusion model to a target domain using only imperfect pseudo-labels. Once trained, it generates high-fidelity, target-aligned images from semantic maps of any synthetic dataset, including low-effort sources created in hours rather than months. The method filters suboptimal generations, rectifies image-label misalignments, and standardises semantics across datasets, transforming weak synthetic data into competitive real-domain training sets. Experiments on five synthetic datasets and two real target datasets show segmentation gains of up to +8.0%pt. mIoU over state-of-the-art translation methods, making rapidly constructed synthetic datasets as effective as high-effort, time-intensive synthetic datasets requiring extensive manual design. This work highlights a valuable collaborative paradigm where fast semantic prototyping, combined with generative models, enables scalable, high-quality training data creation for urban scene understanding.

Figures (7)

• Figure 1: Given a synthetic source dataset (top row) with the corresponding semantic labels (second row), we generate images (third row) that adhere to the semantic map and lie in a particular target distribution, here Cityscapes Cordts2016Cityscapes. Below the images, we report the performance when training a downstream semantic segmentation system NEURIPS2021_64f1f27b on: i) the synthetic source images (Original), ii) using only generated target images from the best performing competitor method zhu2017unpaired, and iii) only our generated images. Being source-agnostic, our approach performs equally well for all synthetic source datasets, regardless of their visual realism, thereby reducing the need for extensive 3D modelling effort and making low-effort datasets like VEIS Saleh_2018_ECCV a viable alternative. It outperforms the synthetic source data and all tested I2I methods by at least +2.0%pt. mIoU.
• Figure 2: Overview of our two-stage training approach: In the first stage, a pre-trained diffusion model is fine-tuned on unlabelled images from the (real) target domain. In the second stage, pseudo-labels predicted by a pre-trained method are used to further fine-tune the diffusion model for semantically-conditioned image synthesis. At test time, semantic maps from any source dataset can be used for the generation of target images.
• Figure 3: Comparison of images translated from UrbanSyn GOMEZ2025130038 to Cityscapes Cordts2016Cityscapes (top) and to ACDC snow sakaridis2021acdc (bottom). Our method features new objects and textures that closely align with the target datasets, while the competitors mostly transform only colours and show many artifacts for complex translations (bottom).
• Figure 4: Generation of images of edge case scenarios from manually created semantic maps with our approach, enabling the quantitative and qualitative analysis of such scenarios and use in training safety-critical systems. Note that these images are not part of Cityscapes Cordts2016Cityscapes.
• Figure 5: Two examples of a mismatch between the generation of our model and the semantic label. A train is incorrectly generated as a "bus" (first column), and a bus is generated as a transport van belonging to the "car" class (second column). If the label is not rectified, this can negatively affect the performance of the downstream model.