Sampling Bag of Views for Open-Vocabulary Object Detection

TL;DR

SBV addresses open-vocabulary object detection by moving beyond a bag of regions to a concept-centered bag of views. It uses an adaptive canvas-based sampling strategy to identify surrounding semantic concepts and forms a bag of concepts, which are represented as three hierarchical views (global, middle, local) and aligned to frozen VLM features via noise and view masks. Representation switching selects the best view per concept, and masks refine the embeddings to suppress background while emphasizing informative views. Empirically, SBV outperforms BARON and other baselines on OV-COCO and OV-LVIS novel-category metrics and achieves substantial CLIP FLOPs reductions, demonstrating both accuracy gains and improved efficiency for open-vocabulary detection.

Abstract

Existing open-vocabulary object detection (OVD) develops methods for testing unseen categories by aligning object region embeddings with corresponding VLM features. A recent study leverages the idea that VLMs implicitly learn compositional structures of semantic concepts within the image. Instead of using an individual region embedding, it utilizes a bag of region embeddings as a new representation to incorporate compositional structures into the OVD task. However, this approach often fails to capture the contextual concepts of each region, leading to noisy compositional structures. This results in only marginal performance improvements and reduced efficiency. To address this, we propose a novel concept-based alignment method that samples a more powerful and efficient compositional structure. Our approach groups contextually related ``concepts'' into a bag and adjusts the scale of concepts within the bag for more effective embedding alignment. Combined with Faster R-CNN, our method achieves improvements of 2.6 box AP50 and 0.5 mask AP over prior work on novel categories in the open-vocabulary COCO and LVIS benchmarks. Furthermore, our method reduces CLIP computation in FLOPs by 80.3% compared to previous research, significantly enhancing efficiency. Experimental results demonstrate that the proposed method outperforms previous state-of-the-art models on the OVD datasets.

Figures (10)

• Figure 1: (a) BARON forms a bag of regions by sampling contextually related neighboring regions for each region proposal. It aggregates region embeddings within a bag into pseudo words in the word embedding space and feeds them to the text encoder. This generates a bag-of-regions embedding aligned with the image feature from the frozen VLMs. (b) Our method groups contextually related "concepts" into a bag. These concepts are then mapped to views at optimal scales, with masks applied to enhance embedding alignment. Finally, our method aligns the bag-of-view embedding with the corresponding features from the frozen VLMs.
• Figure 2: Inefficiency of nearest-neighbor sampling in BARONBARON. Red boxes highlight sampling errors and yellow dotted boxes indicate semantic concepts. (a) BARON increases computational costs by over-sampling empty neighbors. (b) Smaller proposals may miss or cut off co-occurring concept boundaries. (c) Larger region crops introduce noise and reduce the relative size of key objects. Our method addresses these issues by sampling concept windows that effectively capture surrounding concepts.
• Figure 3: Overview of SBV. We call our method SBV (Sampling Bag of Views). (a) SBV probabilistically samples a bag from surrounding semantic concepts detected by generating edges on the canvas, where vertices represent region proposals, edges indicate semantically interrelated visual concepts, and coordinates encode the probabilities of nearby RPN boxes. (b) SBV selectively obtains a representative view for each concept in the bag from three distinct views (e.g. global, middle, and local views) for more effective embedding alignment. These views are further refined using view and noise masks to enhance scene recognition. (c) SBV aligns student and teacher embeddings for the bag of views from pre-trained VLMs.
• Figure 4: Hierarchical views with noise and view masks. (a) Red masks restrict CLIP attention exclusively to the patch areas. (b) Views become brighter as the weight applied increases, transitioning from blue to yellow.
• Figure 5: Qualitative comparisons between SBV and BARON. The images are from COCO’s validation set. Red boxes in the original images highlight the novel categories. The feature maps respond to objects identified by the detector. The novel categories are 'cup’, 'keyboard,' 'elephant,' and 'wine glass’. SBV detects novel objects missed by the baseline.