Adaptive Domain Shift in Diffusion Models for Cross-Modality Image Translation

TL;DR

Cross-modal image translation in diffusion models suffers when global, fixed-domain transfers force sampling through high-energy, off-manifold regions. The authors introduce Cross-Domain Translation SDE (CDTSDE), which embeds domain-shift dynamics directly into the reverse diffusion updates by learning a spatially varying mixing field $\Lambda_t$ and adding a target-consistent restoration drift, enabling on-manifold updates with a continuous-time, exact formulation and a practical first-order sampler. They prove that pixelwise adaptive domain paths strictly lower the path-energy compared to global schedules under mild heterogeneity, and demonstrate consistent improvements in structural fidelity and semantic alignment across MRI (T1↔T2), SAR→Optical, and Electroluminescence→Semantic mapping, with fewer denoising steps. CDTSDE also achieves favorable efficiency, often matching or surpassing baselines at lower sampling counts, and remains compatible with pretrained VP-based diffusion priors, offering a practical pathway for real-world cross-modal synthesis in medicine, remote sensing, and industrial inspection.

Abstract

Cross-modal image translation remains brittle and inefficient. Standard diffusion approaches often rely on a single, global linear transfer between domains. We find that this shortcut forces the sampler to traverse off-manifold, high-cost regions, inflating the correction burden and inviting semantic drift. We refer to this shared failure mode as fixed-schedule domain transfer. In this paper, we embed domain-shift dynamics directly into the generative process. Our model predicts a spatially varying mixing field at every reverse step and injects an explicit, target-consistent restoration term into the drift. This in-step guidance keeps large updates on-manifold and shifts the model's role from global alignment to local residual correction. We provide a continuous-time formulation with an exact solution form and derive a practical first-order sampler that preserves marginal consistency. Empirically, across translation tasks in medical imaging, remote sensing, and electroluminescence semantic mapping, our framework improves structural fidelity and semantic consistency while converging in fewer denoising steps.

Figures (10)

• Figure 1: fixed vs. geometry-aware domain mixture comparison on manifold spaceLeft: The proposed spatially varying schedule $\Lambda_t$ traces a geometry-aware, low-energy path between source and target while preserving endpoint and monotone constraints. Center: One-step schematic—given source $\hat{X}^{\mathrm{src}}_0$ and target $X_0$, the field $\Lambda_t$ (predicted from $\lambda_t$ and position encoding) forms $d_t=\Lambda_t\odot\hat{X}^{\mathrm{src}}_0+(1-\Lambda_t)\odot X_0$ to guide denoising; insets show $\hat{X}^{\mathrm{src}}_0$, $X_0$, and the translated output $\hat{X}^{\mathrm{gen}}_0$. Right: Global linear interpolation $\eta_t\mathbf{1}$ forces a straight path through high-energy regions, leaving larger correction requirement to the diffusion model.
• Figure 2: Visual comparison across three cross-modal tasks: IXI medical conversion, Sentinel SAR-to-optical, and PSCDE semantic mask. (See more in Appendix \ref{['app:visu']}).
• Figure 3: Computational efficiency.
• Figure 4: Additional assessment regarding the robustness to the registration error.
• Figure 5: Visualization of the SAR to Optical image translation task; samples are randomly picked from the testing dataset.

Theorems & Definitions (8)

• Theorem 1
• Proposition 1
• Lemma 1
• Proposition 2: Consistency of the forward marginal
• proof
• Corollary 1
• Proposition 3
• proof