Anatomy-Guided Radiology Report Generation with Pathology-Aware Regional Prompts

TL;DR

This work develops an anatomical region detector that extracts features from distinct anatomical areas, coupled with a novel multi-label lesion detector that identifies global pathologies that outperforms previous state-of-the-art methods on most natural language generation and clinical efficacy metrics.

Abstract

Radiology reporting generative AI holds significant potential to alleviate clinical workloads and streamline medical care. However, achieving high clinical accuracy is challenging, as radiological images often feature subtle lesions and intricate structures. Existing systems often fall short, largely due to their reliance on fixed size, patch-level image features and insufficient incorporation of pathological information. This can result in the neglect of such subtle patterns and inconsistent descriptions of crucial pathologies. To address these challenges, we propose an innovative approach that leverages pathology-aware regional prompts to explicitly integrate anatomical and pathological information of various scales, significantly enhancing the precision and clinical relevance of generated reports. We develop an anatomical region detector that extracts features from distinct anatomical areas, coupled with a novel multi-label lesion detector that identifies global pathologies. Our approach emulates the diagnostic process of radiologists, producing clinically accurate reports with comprehensive diagnostic capabilities. Experimental results show that our model outperforms previous state-of-the-art methods on most natural language generation and clinical efficacy metrics, with formal expert evaluations affirming its potential to enhance radiology practice.

Figures (6)

• Figure 1: Pipeline of the proposed system. Initially, the anatomical region detector identifies and extracts visual features from 29 regions, with the first six shown in the figure. Simultaneously, the multi-label lesion detector identifies global pathologies, assigning multiple lesions to a single bbox. Pathology-aware regional prompts are generated by mapping lesion bboxes to corresponding anatomical regions based on overlap, each tagged with a lesion token. Finally, the report decoder is explicitly guided by both the anatomy-level visual features and the prompt guidance to generate clinically coherent and accurate radiology reports.
• Figure 2: Distribution of the lesion classes showing only the first three and last three classes. Original (42 classes); Modified (21 classes).
• Figure 3: Parent-child relationship in the Chest ImaGenome dataset chestIma, with 'Lung opacity' as the root node. It branches into second-level nodes like 'pleural effusion' and 'lung lesion,' which further split into third-level nodes. The proportion of each node among the 42 lesion classes is indicated in brackets.
• Figure 4: Examples of generated reports by our model and RGRG tanida2023interactive. Green font indicates descriptions consistent with the reference report, while red font denotes incorrect descriptions of negative or unmentioned pathologies. Green highlights confirm correct pathology locations, and red highlights point out error locations.
• Figure 5: Examples of the ablation study. Green font indicates descriptions consistent with the reference report, while red font denotes incorrect descriptions of negative or unmentioned pathologies. Green highlights confirm correct pathology locations, and red highlights point out error locations.