Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Semantic-Enhanced Denoising Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution

Jialu Sui, Xianping Ma, Xiaokang Zhang, Man-On Pun

TL;DR

A novel adaptive diffusion Transformer decoder is developed to bridge the semantic gap between the encoder and decoder through regulating the noise prediction with the global contextual relationships and long-range dependencies in the diffusion process.

Abstract

Remote sensing image super-resolution (SR) is a crucial task to restore high-resolution (HR) images from low-resolution (LR) observations. Recently, the Denoising Diffusion Probabilistic Model (DDPM) has shown promising performance in image reconstructions by overcoming problems inherent in generative models, such as over-smoothing and mode collapse. However, the high-frequency details generated by DDPM often suffer from misalignment with HR images due to the model's tendency to overlook long-range semantic contexts. This is attributed to the widely used U-Net decoder in the conditional noise predictor, which tends to overemphasize local information, leading to the generation of noises with significant variances during the prediction process. To address these issues, an adaptive semantic-enhanced DDPM (ASDDPM) is proposed to enhance the detail-preserving capability of the DDPM by incorporating low-frequency semantic information provided by the Transformer. Specifically, a novel adaptive diffusion Transformer decoder (ADTD) is developed to bridge the semantic gap between the encoder and decoder through regulating the noise prediction with the global contextual relationships and long-range dependencies in the diffusion process. Additionally, a residual feature fusion strategy establishes information exchange between the two decoders at multiple levels. As a result, the predicted noise generated by our approach closely approximates that of the real noise distribution.Extensive experiments on two SR and two semantic segmentation datasets confirm the superior performance of the proposed ASDDPM in both SR and the subsequent downstream applications. The source code will be available at https://github.com/littlebeen/ASDDPM-Adaptive-Semantic-Enhanced-DDPM.

Adaptive Semantic-Enhanced Denoising Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution

TL;DR

A novel adaptive diffusion Transformer decoder is developed to bridge the semantic gap between the encoder and decoder through regulating the noise prediction with the global contextual relationships and long-range dependencies in the diffusion process.

Abstract

Remote sensing image super-resolution (SR) is a crucial task to restore high-resolution (HR) images from low-resolution (LR) observations. Recently, the Denoising Diffusion Probabilistic Model (DDPM) has shown promising performance in image reconstructions by overcoming problems inherent in generative models, such as over-smoothing and mode collapse. However, the high-frequency details generated by DDPM often suffer from misalignment with HR images due to the model's tendency to overlook long-range semantic contexts. This is attributed to the widely used U-Net decoder in the conditional noise predictor, which tends to overemphasize local information, leading to the generation of noises with significant variances during the prediction process. To address these issues, an adaptive semantic-enhanced DDPM (ASDDPM) is proposed to enhance the detail-preserving capability of the DDPM by incorporating low-frequency semantic information provided by the Transformer. Specifically, a novel adaptive diffusion Transformer decoder (ADTD) is developed to bridge the semantic gap between the encoder and decoder through regulating the noise prediction with the global contextual relationships and long-range dependencies in the diffusion process. Additionally, a residual feature fusion strategy establishes information exchange between the two decoders at multiple levels. As a result, the predicted noise generated by our approach closely approximates that of the real noise distribution.Extensive experiments on two SR and two semantic segmentation datasets confirm the superior performance of the proposed ASDDPM in both SR and the subsequent downstream applications. The source code will be available at https://github.com/littlebeen/ASDDPM-Adaptive-Semantic-Enhanced-DDPM.
Paper Structure (22 sections, 16 equations, 11 figures, 9 tables)

This paper contains 22 sections, 16 equations, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Discriminative Models
  4. Generative Models
  5. Methodology
  6. ASDDPM Architecture
  7. Diffusion process
  8. Reverse Process
  9. Conditional noise predictor (CNP)
  10. U-Net Encoder
  11. The proposed dual-decoder Structure
  12. Residual Feature Fusion
  13. Training and Inference
  14. Experiments
  15. Datasets and Metrics
  16. ...and 7 more sections

Figures (11)

  • Figure 1: The architecture of the CNP in the proposed ASDDPM. The four module types, U-Net encoder, Middle step, U-Net decoder, and ADTD, are denoted using different colors and further illustrated using the legends at the right part. The main contributions of the proposed ASDDPM are highlighted and represented by the red dotted box. Moreover, ResBlock means the residual block and ConvBlock contains several convolutional layers to make final channel restoration
  • Figure 2: The process of Inference and Training in our ASDDPM. Trans decoder means ADTD.
  • Figure 3: The architecture of ADTB.
  • Figure 4: The architecture of FI module.
  • Figure 5: The visual comparisons of SR results on the OLI2MSI dataset. (a) LR. (b) HR. (c) RDN. (d) NLSN. (e) TranSMS. (f) SRGAN. (g) ESRGAN. (h) Beby-GAN. (i) Dit. (j) EDiffSR. (k) SRDiff. (l) ASDDPM.
  • ...and 6 more figures