MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detection

TL;DR

Pancreatic cancer remains difficult to diagnose early using radiology due to the organ’s small size and variable appearance. The authors introduce MiniGPT-Pancreas, a cascaded fine-tuning approach based on MiniGPT-v2 that combines a vision encoder with an LLM to perform pancreas detection, pancreas tumor classification, and pancreas tumor detection from CT slices in a conversational interface. The model achieves notable gains in pancreas detection (IoU up to about 0.595 on NIH and 0.550 on MSD) and strong tumor classification performance (accuracy ≈ 0.876, precision ≈ 0.874, recall ≈ 0.878), while tumor localization remains challenging (IoU ≈ 0.168 for MSD tumor detection). Multi-organ evaluation on AbdomenCT-1k demonstrates strong liver grounding (IoU ≈ 0.839) but comparatively lower pancreas localization (IoU ≈ 0.498), underscoring both the potential and the limitations of current MLLMs in pancreas imaging; future work includes 3D visual encoders and cross-modality expansion.

Abstract

Problem: Pancreas radiological imaging is challenging due to the small size, blurred boundaries, and variability of shape and position of the organ among patients. Goal: In this work we present MiniGPT-Pancreas, a Multimodal Large Language Model (MLLM), as an interactive chatbot to support clinicians in pancreas cancer diagnosis by integrating visual and textual information. Methods: MiniGPT-v2, a general-purpose MLLM, was fine-tuned in a cascaded way for pancreas detection, tumor classification, and tumor detection with multimodal prompts combining questions and computed tomography scans from the National Institute of Health (NIH), and Medical Segmentation Decathlon (MSD) datasets. The AbdomenCT-1k dataset was used to detect the liver, spleen, kidney, and pancreas. Results: MiniGPT-Pancreas achieved an Intersection over Union (IoU) of 0.595 and 0.550 for the detection of pancreas on NIH and MSD datasets, respectively. For the pancreas cancer classification task on the MSD dataset, accuracy, precision, and recall were 0.876, 0.874, and 0.878, respectively. When evaluating MiniGPT-Pancreas on the AbdomenCT-1k dataset for multi-organ detection, the IoU was 0.8399 for the liver, 0.722 for the kidney, 0.705 for the spleen, and 0.497 for the pancreas. For the pancreas tumor detection task, the IoU score was 0.168 on the MSD dataset. Conclusions: MiniGPT-Pancreas represents a promising solution to support clinicians in the classification of pancreas images with pancreas tumors. Future research is needed to improve the score on the detection task, especially for pancreas tumors.

Figures (51)

• Figure 1: Architecture of MiniGPT-Pancreas consisting of a vision encoder, a linear projection layer, and an LLM. The visual encoder is fed with each CT slice and one text prompt, e.g. a question. The model then generates text as output, e.g. an answer to a question or the coordinates of a bounding box as text. Adapted from alkhaldi2024minigpt.
• Figure 1: Web interface used for inference, with an example of pancreas detection, tumor classification, and tumor detection. The generated output shows the answer or the coordinates of the bounding box as text. The bounding box is drawn in red on the CT slice.
• Figure 2: JSON data for a specific pancreas slice from the MSD dataset.
• Figure 2: IoU=0.504
• Figure 3: Heatmaps of the GTs and predicted bounding boxes for the pancreas tumor detection task.