The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge

TL;DR

This work tackles misalignment between vision-language models (VLMs) and target concepts in Foundational Few-Shot Object Detection (FSOD). It introduces VLM+, a framework that leverages a multimodal large language model (MM-LLM) to generate multiple referential expressions per category, selecting the best expression via maximum IoU with ground-truth boxes and using these prompts to create high-quality pseudo-labels for training VLMs. The approach integrates two open-set detectors, GLIP and Grounding DINO, and employs iterative pseudo-label optimization with a defined threshold to refine training data and model performance. Empirical results on the final test demonstrate improvements, with Grounding DINO + VLM+ achieving a mAP of $32.56$, and case studies highlight the method’s ability to better align textual prompts with visual concepts, especially for problematic categories. Overall, VLM+ provides a practical path to enhanced FSOD by fusing MM-LLM-driven referential expressions with iterative pseudo-labeling and strong detection backbones.

Abstract

This report introduces an enhanced method for the Foundational Few-Shot Object Detection (FSOD) task, leveraging the vision-language model (VLM) for object detection. However, on specific datasets, VLM may encounter the problem where the detected targets are misaligned with the target concepts of interest. This misalignment hinders the zero-shot performance of VLM and the application of fine-tuning methods based on pseudo-labels. To address this issue, we propose the VLM+ framework, which integrates the multimodal large language model (MM-LLM). Specifically, we use MM-LLM to generate a series of referential expressions for each category. Based on the VLM predictions and the given annotations, we select the best referential expression for each category by matching the maximum IoU. Subsequently, we use these referential expressions to generate pseudo-labels for all images in the training set and then combine them with the original labeled data to fine-tune the VLM. Additionally, we employ iterative pseudo-label generation and optimization to further enhance the performance of the VLM. Our approach achieve 32.56 mAP in the final test.

Figures (4)

• Figure 1: By specifying the interested classes in textual prompts, VLMs can implement zero-shot object detection.
• Figure 2: Poor Alignment Between VLM and Class Prompts. In the nuImages dataset, barriers are defined as road barricades (in red), while the obstacles predicted by the VLMs include roadside steps (in blue).
• Figure 3: The framework of VLM+.
• Figure 4: Visualizing examples: Given referential expressions about categories, VLMs can better detect new entities.