ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue

TL;DR

ZALM3, a Zero-shot strategy to improve vision-language ALignment in Multi-turn Multimodal Medical dialogue, employs an LLM to summarize the keywords from the preceding context and a visual grounding model to extract the regions of interest (RoIs) in the image.

Abstract

The rocketing prosperity of large language models (LLMs) in recent years has boosted the prevalence of vision-language models (VLMs) in the medical sector. In our online medical consultation scenario, a doctor responds to the texts and images provided by a patient in multiple rounds to diagnose her/his health condition, forming a multi-turn multimodal medical dialogue format. Unlike high-quality images captured by professional equipment in traditional medical visual question answering (Med-VQA), the images in our case are taken by patients' mobile phones. These images have poor quality control, with issues such as excessive background elements and the lesion area being significantly off-center, leading to degradation of vision-language alignment in the model training phase. In this paper, we propose ZALM3, a Zero-shot strategy to improve vision-language ALignment in Multi-turn Multimodal Medical dialogue. Since we observe that the preceding text conversations before an image can infer the regions of interest (RoIs) in the image, ZALM3 employs an LLM to summarize the keywords from the preceding context and a visual grounding model to extract the RoIs. The updated images eliminate unnecessary background noise and provide more effective vision-language alignment. To better evaluate our proposed method, we design a new subjective assessment metric for multi-turn unimodal/multimodal medical dialogue to provide a fine-grained performance comparison. Our experiments across three different clinical departments remarkably demonstrate the efficacy of ZALM3 with statistical significance.

Figures (4)

• Figure 1: Framework of the proposed ZALM3. In this example, compared with the original VLM training strategy, our ZALM3 enhances vision-language alignment by extracting the relevant region from images sent by patients in a zero-shot manner, which includes an LLM and a visual grounding model (frozen in the red block). The LLM extracts keywords from the preceding context before the image. The keywords are fed into the visual grounding model to crop the image.
• Figure 2: Percentages of satisfactory/unsatisfactory image quality based on five area ratio segments of RoIs to original images across different clinical departments without and with ZALM3 .
• Figure 3: Visualized inference example from the test set from the department of TCM in our multi-turn multimodal dialogue dataset. Left: conversation between the patient and the real doctor's response as a reference. Middle: Original model response without ZALM3. Right: Model with ZALM3. Green text indicates a satisfactory response (score 3$\sim$4 in Table \ref{['tab:rating']}) in the key location compared with the patient-doctor reference, while red text denotes an unsatisfactory response (score 0$\sim$2 in Table \ref{['tab:rating']}). The gray patches on the patient's eyes are manually added to protect her privacy.
• Figure 4: Illustrated examples from the department of dermatology using GDINO-T and GDINO-B for visual grounding operations using in-context information. The words or phrases with gray color are not activated by GDINO-T or GDINO-B. To protect patients' privacy, the gray patches on patients' eyes are manually added separately when we create this figure, rather than being generated by GDINO models.