Multi-Objective Optimization for Synthetic-to-Real Style Transfer

TL;DR

The paper tackles the domain gap in semantic segmentation caused by synthetic data, by framing style transfer as a sequencing task. It proposes a multi-objective evolutionary approach ( NSGA-II ) to optimize a fixed chain of augmentation operators, evaluated with fast pairwise metrics $DISTS$ and $DreamSim$. Key contributions include the TSP-like pipeline formulation, demonstration of Pareto-front optimization on GTA5 to Cityscapes and ACDC, and insights that optimized pipelines yield diverse trade-offs between content preservation and style alignment. A notable finding is that distributional similarity metrics do not reliably predict downstream segmentation performance, suggesting future work to incorporate segmentation surrogates and compute-time considerations into the objective set.

Abstract

Semantic segmentation networks require large amounts of pixel-level annotated data, which are costly to obtain for real-world images. Computer graphics engines can generate synthetic images alongside their ground-truth annotations. However, models trained on such images can perform poorly on real images due to the domain gap between real and synthetic images. Style transfer methods can reduce this difference by applying a realistic style to synthetic images. Choosing effective data transformations and their sequence is difficult due to the large combinatorial search space of style transfer operators. Using multi-objective genetic algorithms, we optimize pipelines to balance structural coherence and style similarity to target domains. We study the use of paired-image metrics on individual image samples during evolution to enable rapid pipeline evaluation, as opposed to standard distributional metrics that require the generation of many images. After optimization, we evaluate the resulting Pareto front using distributional metrics and segmentation performance. We apply this approach to standard datasets in synthetic-to-real domain adaptation: from the video game GTA5 to real image datasets Cityscapes and ACDC, focusing on adverse conditions. Results demonstrate that evolutionary algorithms can propose diverse augmentation pipelines adapted to different objectives. The contribution of this work is the formulation of style transfer as a sequencing problem suitable for evolutionary optimization and the study of efficient metrics that enable feasible search in this space. The source code is available at: https://github.com/echigot/MOOSS.

Figures (6)

• Figure 1: Overview of the proposed multi-objective pipeline optimization framework. Image transformation pipelines, composed of traditional and deep learning–based operators with a stopping node, are evolved using a multi-objective genetic algorithm. Each pipeline is evaluated using paired-image metrics that measure content preservation (DISTS) and style similarity to the target domain (DreamSim). Ultimately, the evolutionary process produces a Pareto front of non-dominated pipelines representing different trade-offs between both objectives.
• Figure 2: Pairs of images (content, style) used during the optimization process, respectively used for the night, snow, fog, rain, and clear day style transfer. Content images are from the GTA5 dataset. Style images are from the ACDC dataset (night, snow, fog, and rain style) and the Cityscapes dataset (clear day style).
• Figure 3: Populations produced at several stages of the optimization process, from generation 1 on the top left to generation 20 on the bottom right. On the $x$ axis is the DISTS value representing content preservation, and on the $y$ axis the DreamSim value representing style transfer. Both metrics range from 1.0 to 0.0, with lower values being better.
• Figure 4: The final Pareto front of sequences optimized by NSGA-II. The five selected sequences (in red) are highlighted among all optimal solutions (in blue). A reference sequence using only ControlNet is shown in green. The top-left solution corresponds to the highest content preservation (low DISTS), whereas the bottom-right solution achieves the most realistic style (low DreamSim).
• Figure 5: Reference content images, then pipelines 1 $\rightarrow$ 5, and finally ControlNet on the Night (left) and Snow (right) reference style images.