MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queries

TL;DR

This work addresses the gap in multimodal, codemixed Hindi-English medical question summarization by introducing the MMCQS dataset and the MedSumm framework that fuses textual queries with visual cues using LLMs and VLMs. It details data collection from HealthCareMagic, yielding 3015 samples with 18 multimodal symptoms and expert-guided annotations, and demonstrates a three-stage architecture for joint text-visual representation and inference. MedSumm consistently improves automatic and human evaluation metrics over unimodal baselines, with decoder-only LLMs often leading in multimodal scenarios. By enabling richer, image-informed medical question summaries in low-resource multilingual settings, the work paves the way for future enhancements like symptom intensity extraction, video data, and broader language coverage.

Abstract

In the healthcare domain, summarizing medical questions posed by patients is critical for improving doctor-patient interactions and medical decision-making. Although medical data has grown in complexity and quantity, the current body of research in this domain has primarily concentrated on text-based methods, overlooking the integration of visual cues. Also prior works in the area of medical question summarisation have been limited to the English language. This work introduces the task of multimodal medical question summarization for codemixed input in a low-resource setting. To address this gap, we introduce the Multimodal Medical Codemixed Question Summarization MMCQS dataset, which combines Hindi-English codemixed medical queries with visual aids. This integration enriches the representation of a patient's medical condition, providing a more comprehensive perspective. We also propose a framework named MedSumm that leverages the power of LLMs and VLMs for this task. By utilizing our MMCQS dataset, we demonstrate the value of integrating visual information from images to improve the creation of medically detailed summaries. This multimodal strategy not only improves healthcare decision-making but also promotes a deeper comprehension of patient queries, paving the way for future exploration in personalized and responsive medical care. Our dataset, code, and pre-trained models will be made publicly available.

Figures (5)

• Figure 1: Broad categorization of medical disorders in the MMCQS Dataset (MMCQSD). The number of data points corresponding to each category has been provided under each category in the above figure.
• Figure 2: Sample instance of the MMCQS Dataset with the corresponding English query and updated golden summary (target summary).
• Figure 3: Pipeline demonstrating the application of MedSumm for the summarization of multimodal patient queries in a code-mixed environment, aimed at facilitating expedited comprehension of patient queries for doctors and healthcare professionals.
• Figure 4: Model structure of MedSumm showing the different stages of our proposed architecture. The frozen and trainable layers have also been indicated.
• Figure 5: Samples summaries generated by the LLMs in unimodal and multimodal setting. The hallucinated phrases has been shown in red.