Task-Oriented Data Synthesis and Control-Rectify Sampling for Remote Sensing Semantic Segmentation

TL;DR

RS semantic segmentation suffers from labeled-data scarcity. The paper presents TODSynth, combining MM-DiT with unified text–image–mask attention and a Task-guided sampling mechanism called control-rectify flow matching (CRFM) to produce task-oriented synthetic data. Experiments on FUSU-4k and LoveDA-5k show consistent gains over state-of-the-art baselines, especially in few-shot and complex scenes. The work demonstrates the value of integrating architecture-level control with dynamic, task-driven sampling for robust downstream performance in remote sensing.

Abstract

With the rapid progress of controllable generation, training data synthesis has become a promising way to expand labeled datasets and alleviate manual annotation in remote sensing (RS). However, the complexity of semantic mask control and the uncertainty of sampling quality often limit the utility of synthetic data in downstream semantic segmentation tasks. To address these challenges, we propose a task-oriented data synthesis framework (TODSynth), including a Multimodal Diffusion Transformer (MM-DiT) with unified triple attention and a plug-and-play sampling strategy guided by task feedback. Built upon the powerful DiT-based generative foundation model, we systematically evaluate different control schemes, showing that a text-image-mask joint attention scheme combined with full fine-tuning of the image and mask branches significantly enhances the effectiveness of RS semantic segmentation data synthesis, particularly in few-shot and complex-scene scenarios. Furthermore, we propose a control-rectify flow matching (CRFM) method, which dynamically adjusts sampling directions guided by semantic loss during the early high-plasticity stage, mitigating the instability of generated images and bridging the gap between synthetic data and downstream segmentation tasks. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art controllable generation methods, producing more stable and task-oriented synthetic data for RS semantic segmentation.

Figures (5)

• Figure 1: The overall workflow of the task-oriented data synthesis framework (TODSynth) consists of three stages: (a) Training stage using an MM-DiT generative model conditioned on text and mask. (b) Sampling stage with the proposed control-rectify flow matching (CRFM). (c) Downstream tasks trained on a combination of real and synthetic data.
• Figure 2: The architecture of MM-DiT with unified triple attention. For comparison, the mask-adapter and siamese MM-attention schemes are also illustrated.
• Figure 3: Qualitative comparison for mask-to-image generation by various methods.
• Figure 4: Visualization results of pre-synth images with different CRFM steps on the FUSU.
• Figure 5: Visual comparison of generated samples with varying CRFM steps.