Text-guided Zero-Shot Object Localization

TL;DR

A new Zero-Shot Object Localization (ZSOL) framework, which can be guided by prompt words to identify and locate specific objects in an image in the absence of labeled samples, is proposed.

Abstract

Object localization is a hot issue in computer vision area, which aims to identify and determine the precise location of specific objects from image or video. Most existing object localization methods heavily rely on extensive labeled data, which are costly to annotate and constrain their applicability. Therefore, we propose a new Zero-Shot Object Localization (ZSOL) framework for addressing the aforementioned challenges. In the proposed framework, we introduce the Contrastive Language Image Pre-training (CLIP) module which could integrate visual and linguistic information effectively. Furthermore, we design a Text Self-Similarity Matching (TSSM) module, which could improve the localization accuracy by enhancing the representation of text features extracted by CLIP module. Hence, the proposed framework can be guided by prompt words to identify and locate specific objects in an image in the absence of labeled samples. The results of extensive experiments demonstrate that the proposed method could improve the localization performance significantly and establishes an effective benchmark for further research.

Figures (6)

• Figure 1: Paradigm of object localization. (a) Fully Supervised: Costly due to the need for extensive manual labeling. (b) Few-Shot: Highly dependent on the quality of the support samples. (c) Zero-Shot: No annotation, text prompts guide model image understanding.
• Figure 2: Workflow for Zero-Shot Object Localization. (a) The encoder parameters are fixed, and the text self-similarity and visual features are extracted in conjunction with the TSSM module. (b) The density map generated by the model is subjected to post-processing.
• Figure 3: The overall ZSOL framework. (a) Text self-similarity matching. Enhancing the weights of image-related features based on prompt word features. (b) CLIP-based feature extraction. Aligning text self-supporting embedding with image embedding. (c) Density map localization.
• Figure 4: Schematic diagram of TSSM module. Through text self-similarity matching and feature weighting, the recognition capability of the ZSOL model for local regions of interest is enhanced.
• Figure 5: Post-processing steps. By comprehensively applying pooling, filtering, and enhancement techniques to the coordinate points predicted by the density map model, the noise information in the image can be effectively weakened and eliminated.