Dia-LLaMA: Towards Large Language Model-driven CT Report Generation

TL;DR

This work addresses CT report generation by mitigating data imbalance and template bias while handling high-dimensional CT volumes. It presents Dia-LLaMA, which adapts LLaMA2-7B to CT reporting by injecting diagnostic guidance prompts, a ViT3D perceiver visual encoder, and a disease prototype memory bank (DPM) coupled with disease-aware attention (DAA). A diagnostic text prompt (DTP) mechanism converts diagnostic results into textual prompts to steer report generation, and the training objective combines a contrastive disease-prototype loss $\mathcal{L}_{DP}$ with a language modeling loss $\mathcal{L}_{LM}$ as $\mathcal{L} = \mathcal{L}_{DP} + \lambda \mathcal{L}_{LM}$. Evaluations on the CTRG-Chest-548K dataset demonstrate state-of-the-art clinical efficacy and competitive natural language generation metrics, with ablation analyses confirming the benefit of DAA, DPM, and DTP for abnormal-focused reporting and improved handling of rare diseases.

Abstract

Medical report generation has achieved remarkable advancements yet has still been faced with several challenges. First, the inherent imbalance in the distribution of normal and abnormal cases may lead models to exhibit a biased focus on normal samples, resulting in unreliable diagnoses. Second, the frequent occurrence of common template sentences in the reports may overwhelm the critical abnormal information. Moreover, existing works focus on 2D chest X-rays, leaving CT report generation underexplored due to the high-dimensional nature of CT images and the limited availability of CT-report pairs. Recently, LLM has shown a great ability to generate reliable answers with appropriate prompts, which shed light on addressing the aforementioned challenges. In this paper, we propose Dia-LLaMA, a framework to adapt the LLaMA2-7B for CT report generation by incorporating diagnostic information as guidance prompts. Considering the high dimension of CT, we leverage a pre-trained ViT3D with perceiver to extract the visual information. To tailor the LLM for report generation and emphasize abnormality, we extract additional diagnostic information by referring to a disease prototype memory bank, which is updated during training to capture common disease representations. Furthermore, we introduce disease-aware attention to enable the model to adjust attention for different diseases. Experiments on the chest CT dataset demonstrated that our proposed method outperformed previous methods and achieved state-of-the-art on both clinical efficacy performance and natural language generation metrics. The code will be made publically available.

Figures (2)

• Figure 1: The overall architecture. The visual embeddings and diagnostic information are combined into an LLM to generate reports. The disease-aware attention is adopted to extract the disease features, which are used to update the disease prototypes. In inference, the diagnosis results are generated by feature similarity.
• Figure 2: Qualative example of the Baseline and our method. Green indicates the consistent abnormal information, while Red represents the incorrect content.