Recognize Any Regions

TL;DR

RegionSpot tackles open-world region understanding by fusing localization knowledge from SAM with semantic knowledge from CLIP, using a frozen-model cross-attention module to map region tokens to image-wide semantics. It avoids end-to-end fine-tuning, training only a lightweight integration component, and achieves substantial gains over state-of-the-art baselines such as GLIP-L and GroundingDINO-L, while dramatically reducing training time (e.g., ~$3$ million data in a day on $8$ V100 GPUs). The approach yields strong zero-shot performance across LVIS and ODinW benchmarks, with notable improvements for rare categories (e.g., $AP_r$ gains up to $13.1$) and efficient data utilization. This work demonstrates the practicality of combining localization and ViL foundations for open-world region recognition and highlights future potential in leveraging automatic localization capabilities of foundational models for further gains.

Abstract

Understanding the semantics of individual regions or patches of unconstrained images, such as open-world object detection, remains a critical yet challenging task in computer vision. Building on the success of powerful image-level vision-language (ViL) foundation models like CLIP, recent efforts have sought to harness their capabilities by either training a contrastive model from scratch with an extensive collection of region-label pairs or aligning the outputs of a detection model with image-level representations of region proposals. Despite notable progress, these approaches are plagued by computationally intensive training requirements, susceptibility to data noise, and deficiency in contextual information. To address these limitations, we explore the synergistic potential of off-the-shelf foundation models, leveraging their respective strengths in localization and semantics. We introduce a novel, generic, and efficient architecture, named RegionSpot, designed to integrate position-aware localization knowledge from a localization foundation model (e.g., SAM) with semantic information from a ViL model (e.g., CLIP). To fully exploit pretrained knowledge while minimizing training overhead, we keep both foundation models frozen, focusing optimization efforts solely on a lightweight attention-based knowledge integration module. Extensive experiments in open-world object recognition show that our RegionSpot achieves significant performance gain over prior alternatives, along with substantial computational savings (e.g., training our model with 3 million data in a single day using 8 V100 GPUs). RegionSpot outperforms GLIP-L by 2.9 in mAP on LVIS val set, with an even larger margin of 13.1 AP for more challenging and rare categories, and a 2.5 AP increase on ODinW. Furthermore, it exceeds GroundingDINO-L by 11.0 AP for rare categories on the LVIS minival set.

Figures (5)

• Figure 1: Illustration of typical region-level visual understanding architecture. (a) Learning the region recognition model by distilling image-level ViL representations from cropped regions and incorporating them into a detection model ( e.g., gu2021open). (b) Fully fine-tuning both vision and text models with a substantial dataset of region-label pairs. (c) Our proposed approach integrates pretrained (frozen) localization and ViL models, emphasizing the learning of their representational correlation.
• Figure 2: Overview of our proposed RegionSpot. (a) We integrate position-aware tokens from a localization model, such as SAM, with image-level feature maps extracted from a ViL model like CLIP. This integration yields region-level semantic tokens, which are then subjected to region text alignment. (b) Our cross-modal feature interaction design based on the attention mechanism.
• Figure 3: Qualitative prediction results of GLIP-T li2022grounded (first row) and RegionSpot (second row) on the LVIS dataset gupta2019lvis. Our model recognizes the objects more accurately. Best viewed when zooming-in.
• Figure 4: Cross-attention maps in RegionSpot. These maps show that the position-aware token aligns effectively with the semantic feature map of the entire image. In each row, the blue and red boxes are corresponding to the left and right maps respectively.
• Figure 5: More visualizations in comparison with GLIP. Best viewed when zoomed-in.