Enhancing Chest X-ray Classification through Knowledge Injection in Cross-Modality Learning

TL;DR

This work tackles how pre-trained knowledge influences cross-modality learning for Chest X-ray classification by introducing a Set Theory-based framework to generate captions with controllable medical knowledge density. By fine-tuning CLIP on captions that range from coarse labels to fine-grained phenotypic details, the authors demonstrate substantial gains in zero-shot CheXpert classification performance, with 72.5% accuracy using fine-grained injected knowledge versus 49.9% with human captions. The study further shows that knowledge density and domain-specific LLMs meaningfully boost performance, while vision-language models (vLLMs) for caption generation can introduce noise and sometimes underperform specialized setups. These findings highlight the critical role of domain-specific knowledge injection in medical cross-modality learning and suggest broad applicability to other medical imaging tasks and phenotypic descriptions.

Abstract

The integration of artificial intelligence in medical imaging has shown tremendous potential, yet the relationship between pre-trained knowledge and performance in cross-modality learning remains unclear. This study investigates how explicitly injecting medical knowledge into the learning process affects the performance of cross-modality classification, focusing on Chest X-ray (CXR) images. We introduce a novel Set Theory-based knowledge injection framework that generates captions for CXR images with controllable knowledge granularity. Using this framework, we fine-tune CLIP model on captions with varying levels of medical information. We evaluate the model's performance through zero-shot classification on the CheXpert dataset, a benchmark for CXR classification. Our results demonstrate that injecting fine-grained medical knowledge substantially improves classification accuracy, achieving 72.5\% compared to 49.9\% when using human-generated captions. This highlights the crucial role of domain-specific knowledge in medical cross-modality learning. Furthermore, we explore the influence of knowledge density and the use of domain-specific Large Language Models (LLMs) for caption generation, finding that denser knowledge and specialized LLMs contribute to enhanced performance. This research advances medical image analysis by demonstrating the effectiveness of knowledge injection for improving automated CXR classification, paving the way for more accurate and reliable diagnostic tools.

Figures (1)

• Figure 1: Illustration of knowledge injection in cross-modality learning.