DiffCLIP: Few-shot Language-driven Multimodal Classifier

TL;DR

DiffCLIP addresses the challenge of few-shot, language-driven multimodal classification in remote sensing by coupling an unsupervised mask diffusion pretraining stage with a language-supervised fine-tuning phase. It learns a modality-shared encoder that captures cross-modal semantic structure while preserving modality-specific details through dual decoders. Text descriptions are expanded and integrated to enhance semantic alignment between images and class concepts, enabling robust 2-shot and few-shot performance gains over CLIP and other baselines. The approach demonstrates improved accuracy and generalization across multiple high-dimensional remote sensing datasets, highlighting the practical value of diffusion-based pretraining combined with text-guided supervision for domain-specific VLP tasks.

Abstract

Visual language models like Contrastive Language-Image Pretraining (CLIP) have shown impressive performance in analyzing natural images with language information. However, these models often encounter challenges when applied to specialized domains such as remote sensing due to the limited availability of image-text pairs for training. To tackle this issue, we introduce DiffCLIP, a novel framework that extends CLIP to effectively convey comprehensive language-driven semantic information for accurate classification of high-dimensional multimodal remote sensing images. DiffCLIP is a few-shot learning method that leverages unlabeled images for pretraining. It employs unsupervised mask diffusion learning to capture the distribution of diverse modalities without requiring labels. The modality-shared image encoder maps multimodal data into a unified subspace, extracting shared features with consistent parameters across modalities. A well-trained image encoder further enhances learning by aligning visual representations with class-label text information from CLIP. By integrating these approaches, DiffCLIP significantly boosts CLIP performance using a minimal number of image-text pairs. We evaluate DiffCLIP on widely used high-dimensional multimodal datasets, demonstrating its effectiveness in addressing few-shot annotated classification tasks. DiffCLIP achieves an overall accuracy improvement of 10.65% across three remote sensing datasets compared with CLIP, while utilizing only 2-shot image-text pairs. The code has been released at https://github.com/icey-zhang/DiffCLIP.

Figures (5)

• Figure 1: (a) The CLIP model is trained using four randomly sampled image-text pairs from natural image datasets, which are rich in labeled examples.(b) Remote sensing images are annotated in patch format, but there is a severe scarcity of annotated patch samples due to the specialized nature of remote sensing annotations, requiring professional expertise and efficient time management. This significant disparity between datasets makes it challenging to directly apply the CLIP model to remote sensing applications.
• Figure 2: The DiffCLIP framework consists of two main stages: a) Unsupervised Mask Diffusion: A modality-shared image encoder captures consistent features across two modalities, while two modality-specific decoders integrate semantic prompts and unique features. b) Few-shot Language-Driven Classification: DiffCLIP fine-tunes the modality-shared encoder and employs language methods to convey comprehensive semantic information. This approach helps capture rich semantic information inherent in complex data distributions.
• Figure 3: Classification maps of the Trento dataset.
• Figure 4: Classification performance of Houston dataset with different masking ratios and patch size.
• Figure 5: Feature visualization of the visual encoder.