Variational Zero-shot Multispectral Pansharpening

TL;DR

This work proposes a zero-shot pansharpening method by introducing a neural network into the optimization objective, which achieves a similar goal to the so-called deep image prior (DIP) because it implicitly regulates the relationship between the HRMS and PAN images through its inherent structure.

Abstract

Pansharpening aims to generate a high spatial resolution multispectral image (HRMS) by fusing a low spatial resolution multispectral image (LRMS) and a panchromatic image (PAN). The most challenging issue for this task is that only the to-be-fused LRMS and PAN are available, and the existing deep learning-based methods are unsuitable since they rely on many training pairs. Traditional variational optimization (VO) based methods are well-suited for addressing such a problem. They focus on carefully designing explicit fusion rules as well as regularizations for an optimization problem, which are based on the researcher's discovery of the image relationships and image structures. Unlike previous VO-based methods, in this work, we explore such complex relationships by a parameterized term rather than a manually designed one. Specifically, we propose a zero-shot pansharpening method by introducing a neural network into the optimization objective. This network estimates a representation component of HRMS, which mainly describes the relationship between HRMS and PAN. In this way, the network achieves a similar goal to the so-called deep image prior because it implicitly regulates the relationship between the HRMS and PAN images through its inherent structure. We directly minimize this optimization objective via network parameters and the expected HRMS image through iterative updating. Extensive experiments on various benchmark datasets demonstrate that our proposed method can achieve better performance compared with other state-of-the-art methods. The codes are available at https://github.com/xyrui/PSDip.

Figures (13)

• Figure 1: Comparison between supervised, unsupervised and zero-shot multispectral pansharpening methods from the perspective of dataset.
• Figure 2: An overview of the proposed PSDip. We formulate an optimization problem $\min_{\mathcal{X},\theta}~\|\mathcal{Y} - (\mathcal{X}\otimes K)\downarrow_r \|_F^2 + \lambda \|\mathcal{X} - f_\theta(\mathcal{X}, P)\odot\hat{P} \|_F^2$ for multispectral pansharpening. The optimization objective is denoted as $\mathcal{L}(\mathcal{X},\theta)$. The network $f_\theta(\mathcal{X},P)$ takes the HRMS $\mathcal{X}$ and PAN $P$ as inputs and outputs the coefficient tensor $\mathcal{G}$ which comes from the representation $\mathcal{X}=\mathcal{G}\odot\hat{P}$. We choose PanNet PanNet-A-deep-network-architecture-for-pan-sharpening as the backbone of $f_\theta$ (shown in the upper-left part). Before solving the optimization problem, we initialize $f_\theta$ by $\min_\theta ~\|\hat{\mathcal{Y}} - f_\theta(\hat{\mathcal{Y}}, P)\odot(\hat{P}\otimes K)\|$ so that $f_\theta$ predicts a rough $\mathcal{G}$ (shown in the upper-right part). Then, the network parameters and the expected HRMS are formally optimized in the proposed model by alternating minimization (shown in the lower part). Each subproblem is solved by gradient descent based methods. When the optimization finishes after $T$ steps, we derive the expected HRMS $\mathcal{X}_T$ as well as the coefficient tensor $\mathcal{G}_T = f_{\theta_T}(\mathcal{X}_T,P)$.
• Figure 3: Pseudo-color images of HRMS $\mathcal{X}$, the extended PAN $\hat{P}$ and the coefficient $\mathcal{G}$.
• Figure 4: Visualization results on WV2 reduced resolution dataset of all compared methods and the proposed PSDip. Both the restored HRMS and the residual image are shown.
• Figure 5: Visualization results on WV3 reduced resolution dataset of all compared methods and the proposed PSDip. Both the restored HRMS and the residual image are shown.

Theorems & Definitions (2)

• Theorem 1
• proof