UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations

TL;DR

UniDiff addresses the scarcity of pixel-level labels in multimodal remote sensing by adapting a single ImageNet-pretrained diffusion model to HSI and SAR without extensive fine-tuning. It introduces joint timestep–modality FiLM conditioning and pseudo-RGB anchoring to enable parameter-efficient, drift-resistant adaptation, updating only about 5% of parameters. The method operates in two stages: Stage A learns modality-specific representations while Stage B performs dense pixel-wise classification with a frozen backbone and a lightweight head. Across Augsburg and Berlin benchmarks, UniDiff outperforms state-of-the-art baselines in OA, AA, and Kappa, demonstrating effective cross-modal fusion under sparse supervision and robustness to modality variability.

Abstract

Sparse annotations fundamentally constrain multimodal remote sensing: even recent state-of-the-art supervised methods such as MSFMamba are limited by the availability of labeled data, restricting their practical deployment despite architectural advances. ImageNet-pretrained models provide rich visual representations, but adapting them to heterogeneous modalities such as hyperspectral imaging (HSI) and synthetic aperture radar (SAR) without large labeled datasets remains challenging. We propose UniDiff, a parameter-efficient framework that adapts a single ImageNet-pretrained diffusion model to multiple sensing modalities using only target-domain data. UniDiff combines FiLM-based timestep-modality conditioning, parameter-efficient adaptation of approximately 5% of parameters, and pseudo-RGB anchoring to preserve pre-trained representations and prevent catastrophic forgetting. This design enables effective feature extraction from remote sensing data under sparse annotations. Our results with two established multi-modal benchmarking datasets demonstrate that unsupervised adaptation of a pre-trained diffusion model effectively mitigates annotation constraints and achieves effective fusion of multi-modal remotely sensed data.

Figures (5)

• Figure 1: Two-stage parameter-efficient diffusion adaptation for multimodal remote sensing classification using ImageNet-pretrained models. Stage A: Joint adaptation using patch-based FiLM conditioning with pseudo-RGB anchoring. Stage B: Pixel-wise classification on co-registered multimodal features.
• Figure 2: Adaptation benefits across denoising timesteps on the Berlin dataset (Mean F1 score). The adapted HSI+SAR configuration consistently outperforms pretrained baselines, with peak performance observed at timestep 300.
• Figure 3: Visualization on the Berlin dataset with HSI and SAR. (a) Pseudo-RGB HSI, (b) PCA-reduced HSI, (c) SAR image, (d) Sparse Training Labels (limited pixel annotations provided in the benchmark), (e) Ground Truth, (f) UniDiff (ours, multimodal adaptation with HSI + SAR).
• Figure 4: Visualization on the Augsburg dataset. Layout follows \ref{['Berlin_UniDiff_MM_visual_compara']} (a-c) Input modalities, (d) sparse training labels, (e) ground truth, (f) UniDiff results.
• Figure 5: Representative patches generated after joint adaptation with only 5% trainable parameters. Rows show pseudo-RGB (top), PCA (middle), and SAR (bottom). The adapted model preserves modality-specific generative characteristics, demonstrating successful parameter-efficient domain adaptation.