Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two-Stage Random Alternation Framework for One-Shot Pansharpening

Haorui Chen, Zeyu Ren, Jiaxuan Ren, Ran Ran, Jinliang Shao, Jie Huang, Liangjian Deng

TL;DR

TRA-PAN tackles the generalization gap in pansharpening by performing per-instance, one-shot optimization for each MS/PAN pair. It introduces a two-stage pipeline with Stage-1 DAM-based degradation modeling and a warm-up initialization, followed by Stage-2 Random Alternating Optimization that intermittently trains on reduced- and full-resolution data. Empirical results on WV3, QB, and GF2 show superior HQNR and visual fusion quality compared with state-of-the-art methods, demonstrating robustness to real-world conditions. The framework is adaptable to different backbones and offers practical benefits for satellite imagery fusion, with a manageable increase in training time due to the random alternation strategy.

Abstract

Deep learning has substantially advanced pansharpening, achieving impressive fusion quality. However, a prevalent limitation is that conventional deep learning models, which typically rely on training datasets, often exhibit suboptimal generalization to unseen real-world image pairs. This restricts their practical utility when faced with real-world scenarios not included in the training datasets. To overcome this, we introduce a two-stage random alternating framework (TRA-PAN) that performs instance-specific optimization for any given Multispectral(MS)/Panchromatic(PAN) pair, ensuring robust and high-quality fusion. TRA-PAN effectively integrates strong supervision constraints from reduced-resolution images with the physical characteristics of the full-resolution images. The first stage introduces a pre-training procedure, which includes Degradation-Aware Modeling (DAM) to capture spectral degradation mappings, alongside a warm-up procedure designed to reduce training time and mitigate the adverse effects of reduced-resolution data. The second stage employs Random Alternation Optimization (RAO), randomly alternating between reduced- and full-resolution images to refine the fusion model progressively. This adaptive, per-instance optimization strategy, operating in a one-shot manner for each MS/PAN pair, yields superior high-resolution multispectral images. Experimental results demonstrate that TRA-PAN outperforms state-of-the-art (SOTA) methods in quantitative metrics and visual quality in real-world scenarios, underscoring its enhanced practical applicability and robustness.

Two-Stage Random Alternation Framework for One-Shot Pansharpening

TL;DR

TRA-PAN tackles the generalization gap in pansharpening by performing per-instance, one-shot optimization for each MS/PAN pair. It introduces a two-stage pipeline with Stage-1 DAM-based degradation modeling and a warm-up initialization, followed by Stage-2 Random Alternating Optimization that intermittently trains on reduced- and full-resolution data. Empirical results on WV3, QB, and GF2 show superior HQNR and visual fusion quality compared with state-of-the-art methods, demonstrating robustness to real-world conditions. The framework is adaptable to different backbones and offers practical benefits for satellite imagery fusion, with a manageable increase in training time due to the random alternation strategy.

Abstract

Deep learning has substantially advanced pansharpening, achieving impressive fusion quality. However, a prevalent limitation is that conventional deep learning models, which typically rely on training datasets, often exhibit suboptimal generalization to unseen real-world image pairs. This restricts their practical utility when faced with real-world scenarios not included in the training datasets. To overcome this, we introduce a two-stage random alternating framework (TRA-PAN) that performs instance-specific optimization for any given Multispectral(MS)/Panchromatic(PAN) pair, ensuring robust and high-quality fusion. TRA-PAN effectively integrates strong supervision constraints from reduced-resolution images with the physical characteristics of the full-resolution images. The first stage introduces a pre-training procedure, which includes Degradation-Aware Modeling (DAM) to capture spectral degradation mappings, alongside a warm-up procedure designed to reduce training time and mitigate the adverse effects of reduced-resolution data. The second stage employs Random Alternation Optimization (RAO), randomly alternating between reduced- and full-resolution images to refine the fusion model progressively. This adaptive, per-instance optimization strategy, operating in a one-shot manner for each MS/PAN pair, yields superior high-resolution multispectral images. Experimental results demonstrate that TRA-PAN outperforms state-of-the-art (SOTA) methods in quantitative metrics and visual quality in real-world scenarios, underscoring its enhanced practical applicability and robustness.
Paper Structure (23 sections, 9 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 23 sections, 9 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Resolution-Utilization strategies
  4. One-shot learning
  5. Method
  6. Notation
  7. Overview
  8. Stage 1: Pre-training
  9. DAM Procedure
  10. Warm-up Procedure
  11. Stage 2: Random Alternating Optimization
  12. RAO Stage
  13. Reduced-resolution
  14. Full-resolution
  15. Experiments
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Overview of full-resolution training for pansharpening and different paradigms. The upper panel illustrates the pansharpening task, which includes: (a) a depiction of the fundamental spectral and spatial relationships that link MS, PAN, and HRMS images; (b) the spatial degradation pathway from HRMS to low-resolution multispectral (LRMS) images, a process involving the Modulation Transfer Function (MTF) and downsampling; and (c) a demonstration of how spectral relationships between HRMS and PAN are learned through the synthesis of channels within the HRMS (such as the RGB channels shown), thereby illustrating spectral degradation. The lower panel delineates various pansharpening paradigms: (i) a full-resolution training approach; (ii) a reduced-resolution training approach; (iii) a hybrid training approach; and (iv) the proposed TRA-PAN method, which incorporates warm-up and random alternating training stages.
  • Figure 2: The TRA-PAN training framework. The process, illustrated by the top progress bar, initiates with DAM in the pre-training stage, whose network parameters $\theta_D$ are subsequently utilized by the RAO stage. Following DAM, a warm-up procedure operates exclusively on the full-resolution branch. Finally, the RAO stage performs random alternating training. The "R/F Mixed Epoch" is a key part, where the model concurrently trains on both reduced- and full-resolution branches. This process utilizes data from a single input MS/PAN pair to progressively refine the fusion model.
  • Figure 3: The HQNR maps (Top) and visual results (bottom) of all compared approaches on the WV3 full-resolution dataset.
  • Figure 4: Impact of warm-up epochs $m$ on model performance and training dynamics. The left panel illustrates the effect of varying warm-up epochs on the HQNR, alongside $D_\lambda$, $D_s$ metrics. The right panel depicts the relationship between the number of $m$ and the corresponding loss of the fusion network during the subsequent reduced-resolution training in the RAO stage.
  • Figure : Overall TRA-PAN Framework