FSOD-VFM: Few-Shot Object Detection with Vision Foundation Models and Graph Diffusion

TL;DR

This work tackles training-free few-shot object detection by leveraging vision foundation models (UPN for proposals, SAM2 for masks, and DINOv2 for features) to generate and match object proposals with minimal supervision. It introduces FSOD-VFM, which adds a graph-diffusion confidence refinement that propagates scores among proposals to suppress overfragmented, partial-object boxes and emphasize whole-object detections. The approach yields substantial gains across Pascal-5i, COCO-20i, and especially CD-FSOD (e.g., $31.6$ AP in 10-shot on CD-FSOD), demonstrating strong performance without additional training and robust cross-domain generalization. This training-free paradigm, combining universal proposal generation, mask-based RoI features, prototype matching, and graph diffusion, offers a practical, scalable solution for few-shot detection in diverse environments.

Abstract

In this paper, we present FSOD-VFM: Few-Shot Object Detectors with Vision Foundation Models, a framework that leverages vision foundation models to tackle the challenge of few-shot object detection. FSOD-VFM integrates three key components: a universal proposal network (UPN) for category-agnostic bounding box generation, SAM2 for accurate mask extraction, and DINOv2 features for efficient adaptation to new object categories. Despite the strong generalization capabilities of foundation models, the bounding boxes generated by UPN often suffer from overfragmentation, covering only partial object regions and leading to numerous small, false-positive proposals rather than accurate, complete object detections. To address this issue, we introduce a novel graph-based confidence reweighting method. In our approach, predicted bounding boxes are modeled as nodes in a directed graph, with graph diffusion operations applied to propagate confidence scores across the network. This reweighting process refines the scores of proposals, assigning higher confidence to whole objects and lower confidence to local, fragmented parts. This strategy improves detection granularity and effectively reduces the occurrence of false-positive bounding box proposals. Through extensive experiments on Pascal-5$^i$, COCO-20$^i$, and CD-FSOD datasets, we demonstrate that our method substantially outperforms existing approaches, achieving superior performance without requiring additional training. Notably, on the challenging CD-FSOD dataset, which spans multiple datasets and domains, our FSOD-VFM achieves 31.6 AP in the 10-shot setting, substantially outperforming previous training-free methods that reach only 21.4 AP. Code is available at: https://intellindust-ai-lab.github.io/projects/FSOD-VFM.

Figures (5)

• Figure 1: Effect of graph diffusion. The first row shows: (a) an input sample (Figure \ref{['fig:teaser_a']}), (b) its prediction without graph diffusion (Figure \ref{['fig:teaser_b']}), and (c) the final result after applying graph diffusion for 30 steps (Figure \ref{['fig:teaser_c']}). The second row illustrates how the distribution of high-quality boxes (IoU with any ground truth $>$ 0.75) and low-quality boxes (IoU with any ground truth $<$ 0.1) evolves: (d) without graph diffusion (Figure \ref{['fig:teaser_d']}), (e) after 1 step (Figure \ref{['fig:teaser_e']}), and (f) after 30 steps (Figure \ref{['fig:teaser_f']}).
• Figure 2: Overview of FSOD-VFM. Our method integrates UPN, SAM2, and DINOv2 to generate bounding box proposals and perform query matching. We build a graph and perform graph diffusion to mitigate over-fragmentation. The over-fragmented box regions appear more transparent after graph diffusion, indicating that their confidence has decayed.
• Figure 3: Qualitative results of our method on CD-FSOD cdfsod-bench dataset. Our method is shown to achieve precise object detection and classification across varied domains, with the sole requirement of one label per class.
• Figure 4: Segmentation visualization of CD-FSOD cdfsod-bench. SAM2 delivers reliable segmentation results across most scenarios (e.g., insect-related, cartoon, and deep-sea scenes).
• Figure 5: Graph diffusion convergent iteration. The graph diffusion process stop around 70 steps in Pascal-5$^i$everingham2010pascal split1 under 1-shot setting. However, most runs end before 50 steps.