Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement

TL;DR

A novel iterative vision-language representation learning framework designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks.

Abstract

Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision-language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.

Figures (5)

• Figure 1: Our proposed iterative vision-language representation learning framework. In the first iteration, the raw radiology reports are leveraged to gain a general understanding of the patient's condition. In the second stage, the refined reports are employed to further fine-tune the model, guiding the model towards capturing crucial information.
• Figure 2: The two major components of our framework. (a) The vision-language representation learning model with image-text matching determination capability. (b) The key semantic knowledge-emphasized report refinement method.
• Figure 3: Visualizations of phrase grounding with free text on the MS-CXR dataset. (1) to (5) represent five examples in the dataset. White color sentences are the provided free-text annotations. Dashed boxes indicate the annotations outlined by clinical experts. "Ours Iter1" and "Ours Iter2" represent the models trained after the first and second iterations in our framework, respectively.
• Figure 4: Example refined reports of "Ours Iter2 (Claude-3)". The yellow sentences in parentheses represent the supplement sentences via the dictionary matching. The sentences in blue are negative sentences introduced to provide additional information by explicitly stating the absence of these diseases.
• Figure 5: Results of ablation studies by fine-tuning with different ratios of refined reports. The zero-shot classification results on the NIH dataset are reported, including the average scores for 14 diseases (a) and the specific scores for five diseases (b-f).