PneumoLLM: Harnessing the Power of Large Language Model for Pneumoconiosis Diagnosis

TL;DR

PneumoLLM tackles the challenge of diagnosing pneumoconiosis with limited data by eliminating the conventional text-processing branch and reframing the task as a classification problem guided by an LLM. It introduces a contextual multi-token engine to generate diagnostic tokens from image tokens and an information emitter to unidirectionally transfer information from source to diagnosis tokens, all while keeping the vision encoder and LLM largely frozen and training only lightweight adapters and a small classifier. The approach achieves competitive to state-of-the-art performance on a chest radiograph dataset with a data-efficient training footprint, and ablation studies demonstrate the necessity and benefit of the proposed modules, particularly the multi-token engine and information emitter. The work provides a practical pathway for deploying foundation-models in data-scarce medical imaging tasks and suggests avenues for extending the framework to other modalities and multi-label or multi-class scenarios, with code released for reproducibility.

Abstract

The conventional pretraining-and-finetuning paradigm, while effective for common diseases with ample data, faces challenges in diagnosing data-scarce occupational diseases like pneumoconiosis. Recently, large language models (LLMs) have exhibits unprecedented ability when conducting multiple tasks in dialogue, bringing opportunities to diagnosis. A common strategy might involve using adapter layers for vision-language alignment and diagnosis in a dialogic manner. Yet, this approach often requires optimization of extensive learnable parameters in the text branch and the dialogue head, potentially diminishing the LLMs' efficacy, especially with limited training data. In our work, we innovate by eliminating the text branch and substituting the dialogue head with a classification head. This approach presents a more effective method for harnessing LLMs in diagnosis with fewer learnable parameters. Furthermore, to balance the retention of detailed image information with progression towards accurate diagnosis, we introduce the contextual multi-token engine. This engine is specialized in adaptively generating diagnostic tokens. Additionally, we propose the information emitter module, which unidirectionally emits information from image tokens to diagnosis tokens. Comprehensive experiments validate the superiority of our methods and the effectiveness of proposed modules. Our codes can be found at https://github.com/CodeMonsterPHD/PneumoLLM/tree/main.

Figures (6)

• Figure 1: Representative pipelines to elicit knowledge from large models. (a) Traditional works conduct vision-language contrastive learning to align multimodal representations. (b) To utilize large language models, existing works transform images into visual tokens, and send visual tokens to LLM to generate text descriptions. (c) Our work harnesses LLM to diagnose medical images by proper designs, forming a simple and effective pipeline.
• Figure 2: Diagram of the proposed PneumoLLM. The vision encoder processes chest radiography and extracts source tokens. The contextual multi-token engine generates multiple diagnosis tokens conditioned on source tokens. To elicit in-depth knowledge from the LLM, we design the information emitter module within the LLM Transformer layers, enabling unidirectional information flow from source tokens to diagnosis tokens, preserving complete radiographic source details and aggregating critical diagnostic information.
• Figure 3: The illustration examples of dataset preprocessing: two examples labeled as "Normal" and "Pneumoconiosis". The window adjustment opreation use the default window level and width (stored in the DICOM tags) to pre-process the original DICOM files. The segmentation results are obtained using the CheXmask pipeline, as proposed in the paper by gaggion2023chexmask. The selection of the rectangular lung regions is based on the largest external rectangle of the segmentation results.
• Figure 4: Pneumoconiosis diagnosis results comparison with recent prestigious methods. The correct diagnosis results are highlighted in red.
• Figure 5: The t-SNE visualization of feature representation obtained by different networks in comparison experiment.