3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

TL;DR

3MDBench addresses the need for realistic, interactive evaluation of medical dialogue systems by simulating telemedicine consultations with a temperament-based Patient Agent, an LVLM-driven Doctor Agent, and an Assessor Agent. It combines multimodal data (text and images) across 34 diagnoses (2996 images) and enriches contexts with image-associated descriptions to enable richer interactions. The study shows that information-seeking dialogue with internal reasoning improves diagnostic F1 by 6.5%, and incorporating a domain-specific CNN into the LVLM context boosts F1 by up to 20%, highlighting the value of hybrid, context-aware systems. As an open-source benchmark, 3MDBench offers a scalable platform for comparing LVLMs across dialogue strategies, modalities, and patient behaviors, with implications for improving telemedicine AI while acknowledging dataset limitations and ethical considerations.

Abstract

Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM's context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench.

Figures (18)

• Figure 1: The patient's response during the dialogue with the doctor depends on the temperament.
• Figure 2: Agents' interaction pipeline in 3MDBench (left) and an example instance (right). The dialogue $DI_k$ begins with the Patient Agent of sanguine temperament $T_k$, whose first reply contains the complaint image $P_k$ and the basic complaint $BC_k$ highlighted in blue; an additional complaint from the list $AC_k$ appears later in orange, and the final diagnosis $\hat{D}_k$—identified and validated by the Assessor Agent—is shown in green. The Assessor Agent, based on $DI_k$ and $\hat{D}_k$, further provides a structured evaluation of diagnostic performance as well as consultation and communication quality.
• Figure 3: F1 scores by diagnosis categories
• Figure 4: Number of utterances by personality types
• Figure 5: F1 scores by personality types