From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection

TL;DR

The paper addresses data scarcity in multispectral object detection by adapting Vision-Language Models (Grounding DINO and YOLO-World) to RGB-IR inputs, enabling few-shot and open-set capable detection through semantic grounding. It introduces two architectures, MS-GDINO and MS-YOLOW, that fuse RGB and IR features with text prompts via dual-modality backbones and cross-modal decoders, supplemented by adaptive pseudo-labeling to mitigate missing annotations. Across FLIR and M3FD, the approach achieves state-of-the-art performance in few-shot settings and competitive results under full supervision, highlighting the strong cross-spectral transfer of VLM priors. The work demonstrates that semantic grounding, coupled with simple fusion strategies, provides a data-efficient pathway for robust multispectral perception with practical implications for safety-critical systems, and points to future work on zero-shot extension to rare categories.

Abstract

Multispectral object detection is critical for safety-sensitive applications such as autonomous driving and surveillance, where robust perception under diverse illumination conditions is essential. However, the limited availability of annotated multispectral data severely restricts the training of deep detectors. In such data-scarce scenarios, textual class information can serve as a valuable source of semantic supervision. Motivated by the recent success of Vision-Language Models (VLMs) in computer vision, we explore their potential for few-shot multispectral object detection. Specifically, we adapt two representative VLM-based detectors, Grounding DINO and YOLO-World, to handle multispectral inputs and propose an effective mechanism to integrate text, visual and thermal modalities. Through extensive experiments on two popular multispectral image benchmarks, FLIR and M3FD, we demonstrate that VLM-based detectors not only excel in few-shot regimes, significantly outperforming specialized multispectral models trained with comparable data, but also achieve competitive or superior results under fully supervised settings. Our findings reveal that the semantic priors learned by large-scale VLMs effectively transfer to unseen spectral modalities, ofFering a powerful pathway toward data-efficient multispectral perception.

Figures (4)

• Figure 1: Conventional multispectral detectors require many annotated, aligned RGB--IR pairs for training. Our approach leverages the semantic priors of vision--language models (VLMs) to bridge this data gap. Given only a few labeled multispectral examples and a text prompt (e.g., "person, car, bicycle"), the adapted VLM can robustly detect objects across spectral domains by jointly reasoning over visual (RGB) and thermal (IR) cues.
• Figure 2: Overall architecture of MS-X instantiated using either Grounding DINO (MS-GDINO) or YOLO-World (MS-YOLOW). The model processes RGB and IR inputs through separate backbones and encoders, with fusion occurring at the query selection and decoder stages. Text embeddings provide semantic guidance throughout the pipeline.
• Figure 3: Analysis of detection confidence scores from Grounding DINO on few-shot training splits. Top: Histogram of all prediction scores shows a right-skewed distribution with high frequency of low-confidence detections. Bottom: Per-image statistics showing median and 75th percentile scores reveal substantial variation across images, with some having consistently high-confidence predictions while others contain predominantly low-scoring detections.
• Figure 4: Qualitative comparison of object detection results on challenging M3FD scenarios. The top part (a) shows the Ground Truth (GT) annotations on the RGB and IR images (first and second line, respectively). The middle part (b) presents the predictions from the MS-GDINO. The bottom part (c) displays the results from MS-YOLOW method. Models were fine-tuned with 5 shots.