Hyperspectral Image Land Cover Captioning Dataset for Vision Language Models

TL;DR

HyperCap addresses the semantic gap in hyperspectral imaging by creating the first large-scale dataset that pairs pixel-level spectral data with natural language captions. The approach combines four benchmark HSI datasets, patch-based spectral representations, and a hybrid annotation pipeline using LLMs refined by human experts, yielding pixel-precise captions with strong alignment to spectral features. Experiments show that incorporating textual information via diverse fusion strategies substantially improves classification metrics across multiple datasets and also yields competitive captioning performance, with GIT and mPLug excelling in language metrics. This work establishes a foundation for vision-language learning in HSI and opens avenues for retrieval, interpretability, and broader multimodal remote sensing applications, including future scalable captioning models tailored to HSI data.

Abstract

We introduce HyperCap, the first large-scale hyperspectral captioning dataset designed to enhance model performance and effectiveness in remote sensing applications. Unlike traditional hyperspectral imaging (HSI) datasets that focus solely on classification tasks, HyperCap integrates spectral data with pixel-wise textual annotations, enabling deeper semantic understanding of hyperspectral imagery. This dataset enhances model performance in tasks like classification and feature extraction, providing a valuable resource for advanced remote sensing applications. HyperCap is constructed from four benchmark datasets and annotated through a hybrid approach combining automated and manual methods to ensure accuracy and consistency. Empirical evaluations using state-of-the-art encoders and diverse fusion techniques demonstrate significant improvements in classification performance. These results underscore the potential of vision-language learning in HSI and position HyperCap as a foundational dataset for future research in the field.

Figures (42)

• Figure 1: Qualitative Analysis of Class Distribution for the Botswana, Houston13, Indian Pines and KSC datasets.
• Figure 2: Qualitative Analysis of Part-of-Speech Distribution in Captions for the Botswana, Houston13, Indian Pines and KSC datasets.
• Figure 3: Quantitative Visualization of captions per class across Botswana, Houston13, Indian Pines and KSC datasets.
• Figure 4: The plot for the t-SNE visualizations over the Botswana, Houston13, Indian Pines and KSC datasets.
• Figure 5: Visualization of four sample datasets used in the study.