Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidance

Shuchang Ye, Mingyuan Meng, Mingjian Li, Dagan Feng, Jinman Kim

TL;DR

A self-guided segmentation framework (SGSeg) that leverages language guidance for training (multi-modal) while enabling text-free inference (uni-modal), which is the first that enables text-free inference in language-guided segmentation.

Abstract

Segmentation of infected areas in chest X-rays is pivotal for facilitating the accurate delineation of pulmonary structures and pathological anomalies. Recently, multi-modal language-guided image segmentation methods have emerged as a promising solution for chest X-rays where the clinical text reports, depicting the assessment of the images, are used as guidance. Nevertheless, existing language-guided methods require clinical reports alongside the images, and hence, they are not applicable for use in image segmentation in a decision support context, but rather limited to retrospective image analysis after clinical reporting has been completed. In this study, we propose a self-guided segmentation framework (SGSeg) that leverages language guidance for training (multi-modal) while enabling text-free inference (uni-modal), which is the first that enables text-free inference in language-guided segmentation. We exploit the critical location information of both pulmonary and pathological structures depicted in the text reports and introduce a novel localization-enhanced report generation (LERG) module to generate clinical reports for self-guidance. Our LERG integrates an object detector and a location-based attention aggregator, weakly-supervised by a location-aware pseudo-label extraction module. Extensive experiments on a well-benchmarked QaTa-COV19 dataset demonstrate that our SGSeg achieved superior performance than existing uni-modal segmentation methods and closely matched the state-of-the-art performance of multi-modal language-guided segmentation methods.

SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidance

TL;DR

A self-guided segmentation framework (SGSeg) that leverages language guidance for training (multi-modal) while enabling text-free inference (uni-modal), which is the first that enables text-free inference in language-guided segmentation.

Abstract

Segmentation of infected areas in chest X-rays is pivotal for facilitating the accurate delineation of pulmonary structures and pathological anomalies. Recently, multi-modal language-guided image segmentation methods have emerged as a promising solution for chest X-rays where the clinical text reports, depicting the assessment of the images, are used as guidance. Nevertheless, existing language-guided methods require clinical reports alongside the images, and hence, they are not applicable for use in image segmentation in a decision support context, but rather limited to retrospective image analysis after clinical reporting has been completed. In this study, we propose a self-guided segmentation framework (SGSeg) that leverages language guidance for training (multi-modal) while enabling text-free inference (uni-modal), which is the first that enables text-free inference in language-guided segmentation. We exploit the critical location information of both pulmonary and pathological structures depicted in the text reports and introduce a novel localization-enhanced report generation (LERG) module to generate clinical reports for self-guidance. Our LERG integrates an object detector and a location-based attention aggregator, weakly-supervised by a location-aware pseudo-label extraction module. Extensive experiments on a well-benchmarked QaTa-COV19 dataset demonstrate that our SGSeg achieved superior performance than existing uni-modal segmentation methods and closely matched the state-of-the-art performance of multi-modal language-guided segmentation methods.
Paper Structure (18 sections, 2 equations, 5 figures, 2 tables)

This paper contains 18 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Self-guided segmentation framework (SGSeg)
  4. Language-guided U-Net
  5. Localization-enhanced Report Generation
  6. Understanding Text's Role in Language-guided Segmentation
  7. Location-aware pseudo-Label Extraction
  8. Weakly-supervised Localization-enhanced Report Generation
  9. Location-based Attention Aggregation
  10. Experiments
  11. Dataset
  12. Experiment Setup
  13. Implementation Details
  14. Result and Discussion
  15. Comparison with Existing Methods
  16. ...and 3 more sections

Figures (5)

  • Figure 1: The neural network architecture of the proposed SGSeg framework: The blue segment represents the Language-guided U-Net, while the pink segment denotes localization-enhanced report generation processes.
  • Figure 2: Illustration of word importance, where we visualize the attention weight on each token of a report.
  • Figure 3: Comparative Analysis of Segmentation Results: Uni-modal vs. Multi-modal Methods
  • Figure 4: Comparative analysis on the relation between generated text and segmentation outcomes
  • Figure 5: Visualization of the model's attention distribution over the input image arranged sequentially as follows: image, ground truth segmentation, attention map, and attention map projected onto the image