Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation

Songhe Deng, Wei Zhuo, Jinheng Xie, Linlin Shen

TL;DR

Weakly supervised semantic segmentation often suffers from CAM under-activation of target objects and spurious background activation due to limited supervision. QA-CLIMS leverages vision-language pre-training by introducing Question-Answer Prompt Engineering (QAPE) to generate query-adaptive foreground and background text prompts, and a Region Image-Text Contrastive (RITC) network to align region images with open-vocabulary text. The method employs foreground and background contrastive losses, a foreground adaptive thresholding mechanism, and area regularization to produce high-quality CAMs, achieving state-of-the-art results on VOC 2012 and COCO 2014 after refinement. This approach provides a general, query-adaptive WSSS pipeline that exploits rich textual supervision from VLP models to improve object localization and segmentation performance in real-world datasets.

Abstract

Class Activation Map (CAM) has emerged as a popular tool for weakly supervised semantic segmentation (WSSS), allowing the localization of object regions in an image using only image-level labels. However, existing CAM methods suffer from under-activation of target object regions and false-activation of background regions due to the fact that a lack of detailed supervision can hinder the model's ability to understand the image as a whole. In this paper, we propose a novel Question-Answer Cross-Language-Image Matching framework for WSSS (QA-CLIMS), leveraging the vision-language foundation model to maximize the text-based understanding of images and guide the generation of activation maps. First, a series of carefully designed questions are posed to the VQA (Visual Question Answering) model with Question-Answer Prompt Engineering (QAPE) to generate a corpus of both foreground target objects and backgrounds that are adaptive to query images. We then employ contrastive learning in a Region Image Text Contrastive (RITC) network to compare the obtained foreground and background regions with the generated corpus. Our approach exploits the rich textual information from the open vocabulary as additional supervision, enabling the model to generate high-quality CAMs with a more complete object region and reduce false-activation of background regions. We conduct extensive analysis to validate the proposed method and show that our approach performs state-of-the-art on both PASCAL VOC 2012 and MS COCO datasets. Code is available at: https://github.com/CVI-SZU/QA-CLIMS

Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation

TL;DR

Weakly supervised semantic segmentation often suffers from CAM under-activation of target objects and spurious background activation due to limited supervision. QA-CLIMS leverages vision-language pre-training by introducing Question-Answer Prompt Engineering (QAPE) to generate query-adaptive foreground and background text prompts, and a Region Image-Text Contrastive (RITC) network to align region images with open-vocabulary text. The method employs foreground and background contrastive losses, a foreground adaptive thresholding mechanism, and area regularization to produce high-quality CAMs, achieving state-of-the-art results on VOC 2012 and COCO 2014 after refinement. This approach provides a general, query-adaptive WSSS pipeline that exploits rich textual supervision from VLP models to improve object localization and segmentation performance in real-world datasets.

Abstract

Class Activation Map (CAM) has emerged as a popular tool for weakly supervised semantic segmentation (WSSS), allowing the localization of object regions in an image using only image-level labels. However, existing CAM methods suffer from under-activation of target object regions and false-activation of background regions due to the fact that a lack of detailed supervision can hinder the model's ability to understand the image as a whole. In this paper, we propose a novel Question-Answer Cross-Language-Image Matching framework for WSSS (QA-CLIMS), leveraging the vision-language foundation model to maximize the text-based understanding of images and guide the generation of activation maps. First, a series of carefully designed questions are posed to the VQA (Visual Question Answering) model with Question-Answer Prompt Engineering (QAPE) to generate a corpus of both foreground target objects and backgrounds that are adaptive to query images. We then employ contrastive learning in a Region Image Text Contrastive (RITC) network to compare the obtained foreground and background regions with the generated corpus. Our approach exploits the rich textual information from the open vocabulary as additional supervision, enabling the model to generate high-quality CAMs with a more complete object region and reduce false-activation of background regions. We conduct extensive analysis to validate the proposed method and show that our approach performs state-of-the-art on both PASCAL VOC 2012 and MS COCO datasets. Code is available at: https://github.com/CVI-SZU/QA-CLIMS
Paper Structure (28 sections, 11 equations, 7 figures, 6 tables)

This paper contains 28 sections, 11 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Weakly Supervised Semantic Segmentation
  4. Vision-language Pre-training
  5. Contrastive Learning
  6. Method
  7. Overview
  8. Question-Answer Prompt Engineering
  9. Background Questions
  10. Foreground Questions
  11. Post processing
  12. Region Image-Text Contrastive Training
  13. Feature Encoding
  14. Foreground Region Contrastive (FRC) Learning
  15. Background Region Contrastive (BRC) Learning
  16. ...and 13 more sections

Figures (7)

  • Figure 1: (a) CLIMS. Using predefined class-related background text labels. (b) Our QA-CLIMS designed a question-answer prompt engineering procedure to enhance the text-based understanding of image and refine CAMs through contrastive learning on both the foreground target, backgrounds and their corresponding texts.
  • Figure 2: An overview of the proposed QA-CLIMS. Our framework consists of two parts: (a) The Question-Answer Prompt Engineering (QAPE) and (b) the Region Image-Text Contrastive (RITC) network. During the QAPE procedure, we generate a set of image-related semantic texts with pre-defined FG and BG questions and a VQA model. After post-processing the generated texts, we then train the RITC network for CAM. The network is equipped with our foreground adaptive thresholding (FAT) module, and trained through our foreground region contrastive loss $\mathcal{L}_{FRC}$ and our background region contrastive loss $\mathcal{L}_{BRC}$.
  • Figure 3: Visualization of the initial CAMs generated by CAM, CLIMS and the proposed QA-CLIMS. Last row is the text labels used in our QA-CLIMS, where the bold text represents the category label of target object, green for the FG answer and red for the BG answer. Best viewed in color.
  • Figure 4: Per-class CAM quality comparison of different methods, including CLIMS, our baseline (using only FG text containing original category labels), using BG Text and our proposed QA-CLIMS. Evaluation is performed on PASCAL VOC 2012 train set. The numbers above the bars represent the $\Delta$ improvement (%p) of QA-CLIMS compared to CLIMS.
  • Figure 5: Impact of filter ratio on PASCAL VOC 2012.
  • ...and 2 more figures