Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Driven Prompt Learning: A Joint Framework for Multi-modal Cloud Removal and Segmentation

Zaiyan Zhang, Jie Li, Shaowei Shi, Qiangqiang Yuan

TL;DR

This work tackles cloud occlusion by aligning cloud removal with downstream semantic needs. It introduces Task-Driven Prompt Learning for Cloud Removal (TDP-CR), a multimodal framework that uses a Prompt-Guided Fusion block conditioned on a learnable degradation prompt to selectively fuse optical and SAR information. The method employs decoupled optical/SAR encoders, a shared reconstruction decoder, and a lightweight segmentation head trained via a two-phase, parameter-efficient fine-tuning strategy, achieving state-of-the-art PSNR with only 15% of the parameters and improving mIoU by up to 2.6% over strong baselines. By delivering true analysis-ready data, TDP-CR enhances both visual fidelity and semantic utility for Earth observation tasks.

Abstract

Optical remote sensing imagery is indispensable for Earth observation, yet persistent cloud occlusion limits its downstream utility. Most cloud removal (CR) methods are optimized for low-level fidelity and can over-smooth textures and boundaries that are critical for analysis-ready data (ARD), leading to a mismatch between visually plausible restoration and semantic utility. To bridge this gap, we propose TDP-CR, a task-driven multimodal framework that jointly performs cloud removal and land-cover segmentation. Central to our approach is a Prompt-Guided Fusion (PGF) mechanism, which utilizes a learnable degradation prompt to encode cloud thickness and spatial uncertainty. By combining global channel context with local prompt-conditioned spatial bias, PGF adaptively integrates Synthetic Aperture Radar (SAR) information only where optical data is corrupted. We further introduce a parameter-efficient two-phase training strategy that decouples reconstruction and semantic representation learning. Experiments on the LuojiaSET-OSFCR dataset demonstrate the superiority of our framework: TDP-CR surpasses heavy state-of-the-art baselines by 0.18 dB in PSNR while using only 15\% of the parameters, and achieves a 1.4\% improvement in mIoU consistently against multi-task competitors, effectively delivering analysis-ready data.

Task-Driven Prompt Learning: A Joint Framework for Multi-modal Cloud Removal and Segmentation

TL;DR

This work tackles cloud occlusion by aligning cloud removal with downstream semantic needs. It introduces Task-Driven Prompt Learning for Cloud Removal (TDP-CR), a multimodal framework that uses a Prompt-Guided Fusion block conditioned on a learnable degradation prompt to selectively fuse optical and SAR information. The method employs decoupled optical/SAR encoders, a shared reconstruction decoder, and a lightweight segmentation head trained via a two-phase, parameter-efficient fine-tuning strategy, achieving state-of-the-art PSNR with only 15% of the parameters and improving mIoU by up to 2.6% over strong baselines. By delivering true analysis-ready data, TDP-CR enhances both visual fidelity and semantic utility for Earth observation tasks.

Abstract

Optical remote sensing imagery is indispensable for Earth observation, yet persistent cloud occlusion limits its downstream utility. Most cloud removal (CR) methods are optimized for low-level fidelity and can over-smooth textures and boundaries that are critical for analysis-ready data (ARD), leading to a mismatch between visually plausible restoration and semantic utility. To bridge this gap, we propose TDP-CR, a task-driven multimodal framework that jointly performs cloud removal and land-cover segmentation. Central to our approach is a Prompt-Guided Fusion (PGF) mechanism, which utilizes a learnable degradation prompt to encode cloud thickness and spatial uncertainty. By combining global channel context with local prompt-conditioned spatial bias, PGF adaptively integrates Synthetic Aperture Radar (SAR) information only where optical data is corrupted. We further introduce a parameter-efficient two-phase training strategy that decouples reconstruction and semantic representation learning. Experiments on the LuojiaSET-OSFCR dataset demonstrate the superiority of our framework: TDP-CR surpasses heavy state-of-the-art baselines by 0.18 dB in PSNR while using only 15\% of the parameters, and achieves a 1.4\% improvement in mIoU consistently against multi-task competitors, effectively delivering analysis-ready data.
Paper Structure (24 sections, 11 equations, 5 figures, 3 tables)

This paper contains 24 sections, 11 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Prompt and Block Design
  5. Prompt Generator
  6. Prompt-Guided Fusion
  7. Global branch (channel context)
  8. Local branch (spatial prompt attention)
  9. Hybrid attention and fusion
  10. Segmentation Decoder
  11. Multi-scale aggregation
  12. Prediction
  13. Two-phase training strategy
  14. Phase 1: CR pre-training
  15. Phase 2: task-driven fine-tuning
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Overview of TDP-CR. Decoupled encoders extract optical/SAR features, PGF performs prompt-guided adaptive fusion at multiple stages, the shared decoder reconstructs cloud-free imagery, and a lightweight decoder predicts segmentation during Phase 2.
  • Figure 2: Prompt-Guided Fusion (PGF) block combines global channel context and local prompt-conditioned spatial bias for degradation-aware adaptive fusion of optical and SAR features.
  • Figure 3: Visual comparison of cloud removal results. TDP-CR preserves fine details and suppresses artifacts compared to baselines.
  • Figure 4: Visualization of learned degradation prompt maps. We apply Principal Component Analysis (PCA) to project the prompt maps to RGB channels.
  • Figure 5: Visual comparison of land cover segmentation. TDP-CR preserves finer semantic details and boundaries compared to baselines.