See Detail Say Clear: Towards Brain CT Report Generation via Pathological Clue-driven Representation Learning

TL;DR

This work tackles brain CT report generation by addressing redundant visual content and semantic-transfer challenges from limited medical corpora. It introduces Pathological Clue-driven Representation Learning (PCRL), which builds cross-modal representations from segmentation, entity, and theme clues and transfers them to a unified LLM through task-tailored instructions. The framework comprises Segmentation Clue Alignment, Entity Clue Alignment, and Theme Clue Alignment, coupled with a joint training objective that fuses representation learning with report generation. Experiments on the CTRG-Brain dataset demonstrate state-of-the-art performance and show that pathology-guided, multi-level alignment improves both fluency and diagnostic accuracy. The approach advances clinically reliable, coherent brain CT report generation with practical implications for radiology workflows.

Abstract

Brain CT report generation is significant to aid physicians in diagnosing cranial diseases. Recent studies concentrate on handling the consistency between visual and textual pathological features to improve the coherence of report. However, there exist some challenges: 1) Redundant visual representing: Massive irrelevant areas in 3D scans distract models from representing salient visual contexts. 2) Shifted semantic representing: Limited medical corpus causes difficulties for models to transfer the learned textual representations to generative layers. This study introduces a Pathological Clue-driven Representation Learning (PCRL) model to build cross-modal representations based on pathological clues and naturally adapt them for accurate report generation. Specifically, we construct pathological clues from perspectives of segmented regions, pathological entities, and report themes, to fully grasp visual pathological patterns and learn cross-modal feature representations. To adapt the representations for the text generation task, we bridge the gap between representation learning and report generation by using a unified large language model (LLM) with task-tailored instructions. These crafted instructions enable the LLM to be flexibly fine-tuned across tasks and smoothly transfer the semantic representation for report generation. Experiments demonstrate that our method outperforms previous methods and achieves SoTA performance. Our code is available at "https://github.com/Chauncey-Jheng/PCRL-MRG".

Figures (4)

• Figure 1: Comparison of different cross-modal alignment paradigms. (a) Previous way: The origin CT images contain extraneous information unrelated to diagnosis, and separate training for two semantic tasks makes it challenging to find a shared optimal solution, resulting in inadequate cross-modal alignment. (b) Our way: The visual representation is refined to concentrate on pathological clues, and employ joint training with task-tailored instructions via unified LLM to find a transferable representation, leading to better adaption for report generation.
• Figure 2: The overall framework of our method, which mainly consists of an image encoder and a text decoder for brain CT report generation (left). The pathological clue-driven representation learning (right) is proposed to guide the encoder and decoder for more fine-grained representation by three alignment modules: (a) segmentation clue alignment (SCA), (b) entity clue alignment (ECA), and (c) theme clue alignment (TCA).
• Figure 3: Visualization of report generation and mask segmentation. Given the ground truth sample and corresponding entities, the retrieved entity masks are listed in the third column. Reports generated by Baseline, WGAM-HI, and our model are listed in the fourth column. Different colors denote the specific entity words and entity masks, respectively. The English translation is given for a better understanding of the original Chinese reports in CTRG-Brain.
• Figure 4: Visualization of all the segementation masks generated by SAM.