Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge

Longfei Huang, Feng Yu, Zhihao Guan, Zhonghua Wan, Yang Yang

TL;DR

The paper tackles zero-shot referring expression comprehension by exploiting pre-trained vision-language models through a prompt-driven framework. It introduces a tripartite approach: (1) coarse- and fine-grained visual prompts to emphasize target regions, (2) text redundancy reduction to sharpen CLIP’s image-text alignment, and (3) a joint prediction mechanism using aggregation and the Hungarian algorithm to handle multiple descriptions and entries. Key contributions include the multi-granularity prompt strategy, efficient textual denoising, and a robust prediction scheme, achieving substantial gains and first-place performance on the A leaderboard (84.825) and competitive results on the B leaderboard (71.460). The work demonstrates that careful prompt engineering and inference-time joint reasoning can unlock strong zero-shot localization capabilities in multimodal models, providing a practical pathway for zero-shot visual grounding applications.

Abstract

This report presents a solution for the zero-shot referring expression comprehension task. Visual-language multimodal base models (such as CLIP, SAM) have gained significant attention in recent years as a cornerstone of mainstream research. One of the key applications of multimodal base models lies in their ability to generalize to zero-shot downstream tasks. Unlike traditional referring expression comprehension, zero-shot referring expression comprehension aims to apply pre-trained visual-language models directly to the task without specific training. Recent studies have enhanced the zero-shot performance of multimodal base models in referring expression comprehension tasks by introducing visual prompts. To address the zero-shot referring expression comprehension challenge, we introduced a combination of visual prompts and considered the influence of textual prompts, employing joint prediction tailored to the data characteristics. Ultimately, our approach achieved accuracy rates of 84.825 on the A leaderboard and 71.460 on the B leaderboard, securing the first position.

The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge

TL;DR

The paper tackles zero-shot referring expression comprehension by exploiting pre-trained vision-language models through a prompt-driven framework. It introduces a tripartite approach: (1) coarse- and fine-grained visual prompts to emphasize target regions, (2) text redundancy reduction to sharpen CLIP’s image-text alignment, and (3) a joint prediction mechanism using aggregation and the Hungarian algorithm to handle multiple descriptions and entries. Key contributions include the multi-granularity prompt strategy, efficient textual denoising, and a robust prediction scheme, achieving substantial gains and first-place performance on the A leaderboard (84.825) and competitive results on the B leaderboard (71.460). The work demonstrates that careful prompt engineering and inference-time joint reasoning can unlock strong zero-shot localization capabilities in multimodal models, providing a practical pathway for zero-shot visual grounding applications.

Abstract

This report presents a solution for the zero-shot referring expression comprehension task. Visual-language multimodal base models (such as CLIP, SAM) have gained significant attention in recent years as a cornerstone of mainstream research. One of the key applications of multimodal base models lies in their ability to generalize to zero-shot downstream tasks. Unlike traditional referring expression comprehension, zero-shot referring expression comprehension aims to apply pre-trained visual-language models directly to the task without specific training. Recent studies have enhanced the zero-shot performance of multimodal base models in referring expression comprehension tasks by introducing visual prompts. To address the zero-shot referring expression comprehension challenge, we introduced a combination of visual prompts and considered the influence of textual prompts, employing joint prediction tailored to the data characteristics. Ultimately, our approach achieved accuracy rates of 84.825 on the A leaderboard and 71.460 on the B leaderboard, securing the first position.
Paper Structure (13 sections, 3 figures, 1 table)

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Prompt Engineering
  5. Referring Expression Comprehension
  6. Method
  7. Visual Prompt
  8. Removing Redundant Text
  9. Joint Prediction
  10. Experiments
  11. Experiments Setup
  12. Main Results
  13. Conclusion

Figures (3)

  • Figure 1: Referring Expression Comprehension aims to locate the position of a specific object in an image based on a provided textual description, such as accurately locating the proposal of the player in a purple jersey shooting skillfully in the image.
  • Figure 2: Overall Architecture. We use visual prompts to mark regions of interest in images and simultaneously process textual content to enhance the visual-language comprehension capabilities of multimodal models.
  • Figure 3: A summary of the multi-granularity vision prompts used in this paper with the caption "man on left". Note that vision prompts highlighted in pink represent coarse-grained vision prompts, while those highlighted in green represent fine-grained vision prompts. The main difference between coarse-grained and fine-grained vision prompt is the use of SAM to segment objects within the proposal, followed by employing box or circle-based methods to highlight or mask the target object.