PanAdapter: Two-Stage Fine-Tuning with Spatial-Spectral Priors Injecting for Pansharpening

TL;DR

The PanAdapter is proposed, which utilizes additional advanced semantic information from pre-trained models to alleviate the issue of small-scale datasets in pansharpening tasks and can benefit from pre-trained image restoration models and achieve state-of-the-art performance in several benchmark pansharpening datasets.

Abstract

Pansharpening is a challenging image fusion task that involves restoring images using two different modalities: low-resolution multispectral images (LRMS) and high-resolution panchromatic (PAN). Many end-to-end specialized models based on deep learning (DL) have been proposed, yet the scale and performance of these models are limited by the size of dataset. Given the superior parameter scales and feature representations of pre-trained models, they exhibit outstanding performance when transferred to downstream tasks with small datasets. Therefore, we propose an efficient fine-tuning method, namely PanAdapter, which utilizes additional advanced semantic information from pre-trained models to alleviate the issue of small-scale datasets in pansharpening tasks. Specifically, targeting the large domain discrepancy between image restoration and pansharpening tasks, the PanAdapter adopts a two-stage training strategy for progressively adapting to the downstream task. In the first stage, we fine-tune the pre-trained CNN model and extract task-specific priors at two scales by proposed Local Prior Extraction (LPE) module. In the second stage, we feed the extracted two-scale priors into two branches of cascaded adapters respectively. At each adapter, we design two parameter-efficient modules for allowing the two branches to interact and be injected into the frozen pre-trained VisionTransformer (ViT) blocks. We demonstrate that by only training the proposed LPE modules and adapters with a small number of parameters, our approach can benefit from pre-trained image restoration models and achieve state-of-the-art performance in several benchmark pansharpening datasets. The code will be available soon.

Figures (7)

• Figure 1: PanAdapter's two-stage fine-tuning framework.
• Figure 2: Network structure of the first stage, i.e., the Local Prior Extraction Stage, and the details about the Local Prior Extraction (LPE) block. The frozen CNN networks are pre-trained EDSR lim2017enhanced.
• Figure 3: Network structure of the second stage, i.e., the Multiscale Feature Interaction Stage, and the details about the Cascade Token Fusioner (CTF) and the Cascade Token Injector (CTI). The Spatial-Spectral Priors Extraction Network (SSPEN) and the Multiscale Feature Interaction Stage (MFIN) denote the pre-trained and fine-tuning networks of the first stage and the second stage, respectively. The frozen ViT is pre-trained Image Processing Transformer (IPT) chen2021pre.
• Figure 4: Qualitative evaluation result comparisons with previous pansharpening methods on GF2 reduced-resolution dataset. The first row consists of natural color output, while the second row presents the absolute error maps.
• Figure 5: Qualitative evaluation result comparisons with previous pansharpening methods on WV3 reduced-resolution dataset.