Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Weakly Supervised Semantic Segmentation with Multi-modal Foundation Models: An End-to-End Approach

Elham Ravanbakhsh, Cheng Niu, Yongqing Liang, J. Ramanujam, Xin Li

TL;DR

This paper tackles the high cost of pixel-level annotations in semantic segmentation by introducing an end-to-end weakly supervised framework that leverages foundation models. By applying SAM inside object bounding boxes, together with Grounding-DINO for localization and CLIP for image-label prediction, the approach generates high-quality, class-aware pseudo-labels without image-level supervision, which are then used to train a standard segmentation model. The method achieves state-of-the-art results on PASCAL VOC 2012 and MS COCO 2014, with substantial improvements in pseudo-label quality over CAM- and SAM-based baselines (e.g., improvements of 9.55% and 40.87% in specific comparisons). Overall, this work reduces labeling burden while delivering finer-grained segmentation boundaries, with strong practical impact for real-world applications.

Abstract

Semantic segmentation is a core computer vision problem, but the high costs of data annotation have hindered its wide application. Weakly-Supervised Semantic Segmentation (WSSS) offers a cost-efficient workaround to extensive labeling in comparison to fully-supervised methods by using partial or incomplete labels. Existing WSSS methods have difficulties in learning the boundaries of objects leading to poor segmentation results. We propose a novel and effective framework that addresses these issues by leveraging visual foundation models inside the bounding box. Adopting a two-stage WSSS framework, our proposed network consists of a pseudo-label generation module and a segmentation module. The first stage leverages Segment Anything Model (SAM) to generate high-quality pseudo-labels. To alleviate the problem of delineating precise boundaries, we adopt SAM inside the bounding box with the help of another pre-trained foundation model (e.g., Grounding-DINO). Furthermore, we eliminate the necessity of using the supervision of image labels, by employing CLIP in classification. Then in the second stage, the generated high-quality pseudo-labels are used to train an off-the-shelf segmenter that achieves the state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014.

Enhancing Weakly Supervised Semantic Segmentation with Multi-modal Foundation Models: An End-to-End Approach

TL;DR

This paper tackles the high cost of pixel-level annotations in semantic segmentation by introducing an end-to-end weakly supervised framework that leverages foundation models. By applying SAM inside object bounding boxes, together with Grounding-DINO for localization and CLIP for image-label prediction, the approach generates high-quality, class-aware pseudo-labels without image-level supervision, which are then used to train a standard segmentation model. The method achieves state-of-the-art results on PASCAL VOC 2012 and MS COCO 2014, with substantial improvements in pseudo-label quality over CAM- and SAM-based baselines (e.g., improvements of 9.55% and 40.87% in specific comparisons). Overall, this work reduces labeling burden while delivering finer-grained segmentation boundaries, with strong practical impact for real-world applications.

Abstract

Semantic segmentation is a core computer vision problem, but the high costs of data annotation have hindered its wide application. Weakly-Supervised Semantic Segmentation (WSSS) offers a cost-efficient workaround to extensive labeling in comparison to fully-supervised methods by using partial or incomplete labels. Existing WSSS methods have difficulties in learning the boundaries of objects leading to poor segmentation results. We propose a novel and effective framework that addresses these issues by leveraging visual foundation models inside the bounding box. Adopting a two-stage WSSS framework, our proposed network consists of a pseudo-label generation module and a segmentation module. The first stage leverages Segment Anything Model (SAM) to generate high-quality pseudo-labels. To alleviate the problem of delineating precise boundaries, we adopt SAM inside the bounding box with the help of another pre-trained foundation model (e.g., Grounding-DINO). Furthermore, we eliminate the necessity of using the supervision of image labels, by employing CLIP in classification. Then in the second stage, the generated high-quality pseudo-labels are used to train an off-the-shelf segmenter that achieves the state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014.
Paper Structure (22 sections, 4 figures, 6 tables)

This paper contains 22 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CAM-based Pseudo-labels
  4. Foundation model assisted Pseudo-labels
  5. Method
  6. Overview
  7. Pseudo-label Generation Module
  8. Language Image Matching
  9. Bounding Box Detection
  10. Pseudo-label Generation inside the Bounding Box
  11. Segmentation Module
  12. Experiments
  13. Datasets and Evaluation Metrics
  14. Implementation details
  15. Comparison to the State-of-the-Art
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Illustration of the partial and false activation issues of CAMs on PASCAL VOC 2012 train set, our result (based on segmentation inside the bounding box), and the ground-truth. Row 1 presents the partial activation of CAMs while Row 2 demonstartes the false activation.
  • Figure 2: Overview of the proposed two-stage WSSS training pipeline: (1) Generation of pseudo labels aided by foundation models (green box), and (2) Training of a segmentation network using these pseudo labels (purple box). During the inference phase, the trained segmenter is used to directly segment the input image.
  • Figure 3: The pseudo-labels generated by our proposed framework and comparison between WSSS-SAM and CLIP-ES on PASCAL VOC 2012 training set.
  • Figure 4: Visualization of the final segmentation results, CLIP-ES, and WSSS-SAM on MS COCO 2014 validation set.