Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Multimodal Models for Enhanced Neuroimaging Diagnostics in Alzheimer's Disease

Francesco Chiumento, Mingming Liu

TL;DR

This paper generates synthetic diagnostic reports using GPT-4o-mini on structured data from the OASIS-4 dataset, which comprises 663 patients, and generates neurological reports directly from the images in the dataset leveraging the pre-trained BiomedCLIP and T5 models.

Abstract

The rapid advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown great potential in medical diagnostics, particularly in radiology, where datasets such as X-rays are paired with human-generated diagnostic reports. However, a significant research gap exists in the neuroimaging field, especially for conditions such as Alzheimer's disease, due to the lack of comprehensive diagnostic reports that can be utilized for model fine-tuning. This paper addresses this gap by generating synthetic diagnostic reports using GPT-4o-mini on structured data from the OASIS-4 dataset, which comprises 663 patients. Using the synthetic reports as ground truth for training and validation, we then generated neurological reports directly from the images in the dataset leveraging the pre-trained BiomedCLIP and T5 models. Our proposed method achieved a BLEU-4 score of 0.1827, ROUGE-L score of 0.3719, and METEOR score of 0.4163, revealing its potential in generating clinically relevant and accurate diagnostic reports.

Leveraging Multimodal Models for Enhanced Neuroimaging Diagnostics in Alzheimer's Disease

TL;DR

This paper generates synthetic diagnostic reports using GPT-4o-mini on structured data from the OASIS-4 dataset, which comprises 663 patients, and generates neurological reports directly from the images in the dataset leveraging the pre-trained BiomedCLIP and T5 models.

Abstract

The rapid advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown great potential in medical diagnostics, particularly in radiology, where datasets such as X-rays are paired with human-generated diagnostic reports. However, a significant research gap exists in the neuroimaging field, especially for conditions such as Alzheimer's disease, due to the lack of comprehensive diagnostic reports that can be utilized for model fine-tuning. This paper addresses this gap by generating synthetic diagnostic reports using GPT-4o-mini on structured data from the OASIS-4 dataset, which comprises 663 patients. Using the synthetic reports as ground truth for training and validation, we then generated neurological reports directly from the images in the dataset leveraging the pre-trained BiomedCLIP and T5 models. Our proposed method achieved a BLEU-4 score of 0.1827, ROUGE-L score of 0.3719, and METEOR score of 0.4163, revealing its potential in generating clinically relevant and accurate diagnostic reports.

Paper Structure

This paper contains 23 sections, 9 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Current Advancements in LLMs and VLMs
  4. Challenges of AI Models in Neuroimaging
  5. Relevant Datasets
  6. OASIS Datasets
  7. OASIS-1
  8. OASIS-2
  9. OASIS-3
  10. OASIS-4
  11. Methodology
  12. Preprocessing of Structured Data
  13. Generation of Synthetic Diagnostic Reports
  14. Preprocessing of T1-Weighted MR Images
  15. Training and Validation of the Models
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Data preparation workflow for structured data, MR images, and segmentation volumes.
  • Figure 2: Overview of the methodology, showing data preprocessing, model fine-tuning, and evaluation phases.
  • Figure 3: Boxplot comparison of BLEU, ROUGE, and METEOR evaluation metrics for T5-Small, T5-Base, and T5-Large models on the OASIS-4 dataset.
  • Figure 4: Generation of a textual report using BiomedCLIP and T5-large and comparison with the ground truth.