Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance

TL;DR

Ultrasound report generation faces cross-modality feature gaps and the need for long, detailed text. The authors propose a three-component framework comprising an unsupervised Knowledge Distiller to extract prior textual knowledge from reports, a Knowledge Matched Visual Extractor to align image features with this knowledge, and a Transformer-based Report Generator augmented by a Similarity Comparer to enforce global semantic consistency. Evaluated on three large multi-organ ultrasound datasets (breast, thyroid, liver), the approach consistently outperforms strong baselines across NLG and clinically oriented metrics, while maintaining efficiency. The work advances cross-modal medical report generation by leveraging unsupervised textual priors to guide visual feature learning, though it acknowledges limitations in handling size/ location/ count terms due to long-tail vocabulary distributions and points to future improvements in fine-grained descriptions.

Abstract

Automatic report generation has arisen as a significant research area in computer-aided diagnosis, aiming to alleviate the burden on clinicians by generating reports automatically based on medical images. In this work, we propose a novel framework for automatic ultrasound report generation, leveraging a combination of unsupervised and supervised learning methods to aid the report generation process. Our framework incorporates unsupervised learning methods to extract potential knowledge from ultrasound text reports, serving as the prior information to guide the model in aligning visual and textual features, thereby addressing the challenge of feature discrepancy. Additionally, we design a global semantic comparison mechanism to enhance the performance of generating more comprehensive and accurate medical reports. To enable the implementation of ultrasound report generation, we constructed three large-scale ultrasound image-text datasets from different organs for training and validation purposes. Extensive evaluations with other state-of-the-art approaches exhibit its superior performance across all three datasets. Code and dataset are valuable at this link.

Figures (10)

• Figure 1: Examples of the ultrasound report and radiology report. The original ultrasound report is written in Chinese.
• Figure 2: An overview of our proposed report generation framework. The orange section shows the Knowledge Distiller (KD), which extracts potential prior knowledge from ultrasound reports using unsupervised learning methods. The blue section is the Knowledge Matched Visual Extractor (KMVE), which uses prior knowledge extracted by the KD module to guide the visual extractor to capture knowledge-related visual features, addressing the problem of mismatch between visual and textual features. The green section shows the Report Generator (RG), which generates a text sequence from visual features, with a Transformer Encoder Decoder backbone and a proposed Similarity Comparer module.
• Figure 3: Age and gender distribution of our collected ultrasound datasets from three organs.
• Figure 4: Hyper-parameter searching with 10% liver training data. (a) shows the report generation performance evaluated by the ROUGE-L metric, whereas (b) shows the results evaluated by the METEOR metric.
• Figure 5: Heatmap of Clustering Results with different Dimensionality Reduction and Cluster Numbers. Each heatmap in this table displays clustering results, with the x-axis representing the dimensions of dimensionality reduction and the y-axis indicating different numbers of clusters. The values in each cell of the heatmap represent the silhouette coefficient scores, which reflect the performance of the clustering for each combination of dimension reduction and cluster numbers.