Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation

TL;DR

A co-evolutionary abnormality detection and report generation (CoE-DG) framework that utilizes both fully labeled and weakly labeled data to achieve mutual promotion between the abnormality detection and report generation tasks and introduces a bi-directional information interaction strategy with generator-guided information propagation (GIP) and detector-guided information propagation (DIP).

Abstract

Anatomical abnormality detection and report generation of chest X-ray (CXR) are two essential tasks in clinical practice. The former aims at localizing and characterizing cardiopulmonary radiological findings in CXRs, while the latter summarizes the findings in a detailed report for further diagnosis and treatment. Existing methods often focused on either task separately, ignoring their correlation. This work proposes a co-evolutionary abnormality detection and report generation (CoE-DG) framework. The framework utilizes both fully labeled (with bounding box annotations and clinical reports) and weakly labeled (with reports only) data to achieve mutual promotion between the abnormality detection and report generation tasks. Specifically, we introduce a bi-directional information interaction strategy with generator-guided information propagation (GIP) and detector-guided information propagation (DIP). For semi-supervised abnormality detection, GIP takes the informative feature extracted by the generator as an auxiliary input to the detector and uses the generator's prediction to refine the detector's pseudo labels. We further propose an intra-image-modal self-adaptive non-maximum suppression module (SA-NMS). This module dynamically rectifies pseudo detection labels generated by the teacher detection model with high-confidence predictions by the student.Inversely, for report generation, DIP takes the abnormalities' categories and locations predicted by the detector as input and guidance for the generator to improve the generated reports.

Figures (9)

• Figure 1: Overview of the proposed framework. NMS: non-maximum suppression; GIP: generator-guided information propagation; DIP: detector-guided information propagation. For semi-supervised abnormality detection, a self-adaptive NMS module dynamically rectifies pseudo detection labels generated by the teacher detection model $F^I_t$ with high-confidence predictions by the student $F^I_s$. The GIP takes the feature extracted by the generator $F^R$ as an auxiliary input to $F^I_s$; it also uses $F^R$’s prediction to refine the pseudo labels further. Inversely, for report generation, the DIP takes the abnormalities detected by $F^I_s$ as input and guidance to $F^R$ to improve generated reports. The abnormalities' categories predicted by $F^I_s$ are also used to supervise $F^R$'s training via $\mathcal{L}^R_{cls}$ for weakly labeled samples.
• Figure 2: Illustration of the co-evolution strategy. "D" and "G" represent detector-guided and generator-guided information propagation (DIP and GIP), respectively. The $k$th iteration student detection model $F_{s,k}^I$ is distilled from the teacher $F_{t,k-1}^I$ guided by the generation model $F_{k}^R$ via GIP. Subsequently, $F_{s,k}^I$ is frozen and used to 1) guide the training of the $(k+1)$th generation model $F_{k+1}^R$ via DIP, and 2) serve as the teacher detection model in the next iteration, i.e., $F_{s,k}^I\rightarrow F_{t, k}^I$.
• Figure 3: Performance of the detection and report generation models as a function of iterations on the validation set of MS-CXR.
• Figure 4: Visualization of the detection results by Soft Teacher xu2021end, GLIPv2 zhang2022glipv2 and ours. The green and red boxes are the ground truth (GT) and predictions, respectively. The texts below images indicate abnormality categories of the boxes from left to right, and "-" indicates a failure to detect an abnormality.
• Figure 5: Example reports generated by R2Genchen2020generating, GLIPv2zhang2022glipv2, our method, and the ground truth (GT). Texts highlighted with background colors are abnormalities described in GT. Texts in red indicate false positives, and those in other colors indicate false negatives.