WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models

TL;DR

The paper addresses multimodal dermatology question answering within MEDIQA-M3G by evaluating two English submissions: a two-pass Claude 3 Opus API pipeline and a CLIP-style image–disease embedding trained for image classification. The former uses sequential LLM reasoning to first generate differential diagnoses from images and then output the final disease name, achieving 1st place, while the latter builds a joint image–label embedding for nearest-neighbor disease classification, achieving 2nd place. The study demonstrates that multi-stage LLM frameworks and CLIP-inspired embeddings can yield strong results in challenging multimodal medical tasks, though overall deltaBLEU scores remain modest and reproducibility and metric biases warrant careful interpretation. The work provides practical insights into prompt design, retrieval strategies, batch-size effects, and post-processing tricks, offering concrete directions for improving clinical multimodal answer generation in the near term.

Abstract

This paper outlines our submission to the MEDIQA2024 Multilingual and Multimodal Medical Answer Generation (M3G) shared task. We report results for two standalone solutions under the English category of the task, the first involving two consecutive API calls to the Claude 3 Opus API and the second involving training an image-disease label joint embedding in the style of CLIP for image classification. These two solutions scored 1st and 2nd place respectively on the competition leaderboard, substantially outperforming the next best solution. Additionally, we discuss insights gained from post-competition experiments. While the performance of these two solutions have significant room for improvement due to the difficulty of the shared task and the challenging nature of medical visual question answering in general, we identify the multi-stage LLM approach and the CLIP image classification approach as promising avenues for further investigation.

Figures (3)

• Figure 1: Overview of the two winning solutions. A) Test cases are directly submitted to the Claude 3 Opus API. The first of the two consecutive API calls generates differential diagnosis using only the images in the test cases and the second API call optionally includes the associated queries, specifies formatting, and generates final answer. B) The medical discussions included as a part of the training data is used to extract the most likely disease label for each case using GPT4-Turbo from OpenAI. The resulting image-disease label pair are used in conjunction with publicly available data to train a joint embedding in the style of CLIP. The disease labels are embedded using EmbeddingV3 from OpenAI and used to train the image encoder (ResNet50) and both the image and text projection layers. Finally, once the model is trained, the test images are classified inside the learned joint embedding which becomes the final output before performing post processing.
• Figure S1: Representative case example illustrating the images of the skin condition, their associated textual query and the predicted response given.
• Figure S2: PCA visualization of all the training disease labels embedded by the EmbeddingV3 model. Skin conditions that are semantically similar are clustered together in this representation space.