Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights

TL;DR

This work interrogates whether open LVLMs can perform multimodal carotid plaque risk stratification by fusing ultrasound images with structured clinical data. In zero-shot tests, some models reliably detect ultrasound modality and carotid anatomy but fail to predict stroke risk, highlighting gaps in domain priors. Adapting LLaVa-NeXT-Vicuna to ultrasound via LoRA and incorporating tabular data yields competitive AUC with CNN baselines and improves specificity, illustrating the value of multimodal integration and domain adaptation for clinical translation. The findings imply that LVLMs hold promise for scalable, interactive cardiovascular decision support, but robust clinical deployment will require larger, balanced datasets, temporal ultrasound information, calibration improvements, and interpretable reasoning workflows.

Abstract

Reliable risk assessment for carotid atheromatous disease remains a major clinical challenge, as it requires integrating diverse clinical and imaging information in a manner that is transparent and interpretable to clinicians. This study investigates the potential of state-of-the-art and recent large vision-language models (LVLMs) for multimodal carotid plaque assessment by integrating ultrasound imaging (USI) with structured clinical, demographic, laboratory, and protein biomarker data. A framework that simulates realistic diagnostic scenarios through interview-style question sequences is proposed, comparing a range of open-source LVLMs, including both general-purpose and medically tuned models. Zero-shot experiments reveal that even if they are very powerful, not all LVLMs can accurately identify imaging modality and anatomy, while all of them perform poorly in accurate risk classification. To address this limitation, LLaVa-NeXT-Vicuna is adapted to the ultrasound domain using low-rank adaptation (LoRA), resulting in substantial improvements in stroke risk stratification. The integration of multimodal tabular data in the form of text further enhances specificity and balanced accuracy, yielding competitive performance compared to prior convolutional neural network (CNN) baselines trained on the same dataset. Our findings highlight both the promise and limitations of LVLMs in ultrasound-based cardiovascular risk prediction, underscoring the importance of multimodal integration, model calibration, and domain adaptation for clinical translation.

Figures (4)

• Figure 1: Overview of the methods that were applied in this study. 1) A zero-shot evaluation framework was designed to assess state-of-the-art large vision language models in i) medical reasoning and ii) risk stratification using single and multimodal inputs from carotid ultrasound images and clinical, demographic, laboratory, and protein analysis tabular data. 2) The adaptation of LLaVA-NeXT is performed using Low-rank Adaptation (LoRA) and 3-fold cross validation in a single and multimodal setting.
• Figure 2: An example of the context-free interview process. The model is prompted with the image and the interview-style text. Each model produces different responses, indicating its zero-shot capabilities in medical reasoning.
• Figure 3: An example of the imaging context (single modal) prompt. The model is prompted with the image and user text that provides context for the image and asks a question that should be answered with a single word. This way of prompting is designed to assess the model's diagnosis capabilities from a single image.
• Figure 4: An example of the imaging and tabular (multimodal) context prompt. The model is prompted with the image and user text that provides context for the image, demographic, clinical, and laboratory information, inside the test and asks a question that should be answered with a single word. This way of prompting is designed to assess the model's diagnosis capabilities from a multimodal prompt.