Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PulseMind: A Multi-Modal Medical Model for Real-World Clinical Diagnosis

Jiao Xu, Junwei Liu, Jiangwei Lao, Qi Zhu, Yunpeng Zhao, Congyun Jin, Shinan Liu, Zhihong Lu, Lihe Zhang, Xin Chen, Jian Wang, Ping Wang

TL;DR

PulseMind tackles the gap in real-world clinical diagnostics by integrating a large-scale multi-turn, multi-modal dataset (MediScope), a dedicated diagnostic benchmark (PulseMind Benchmark), and a relative-reward reinforcement learning framework (CRPO). It demonstrates strong performance on both the PulseMind Benchmark and 11 public medical QA benchmarks, indicating robust generalization across modalities and tasks. The approach emphasizes realistic data collection, multi-turn context, and human-aligned optimization, offering a practical foundation for clinical diagnostic dialogue AI. These advances have potential to improve diagnostic efficiency and consistency in real-world clinical settings where heterogeneous inputs and physician-patient interactions are common.

Abstract

Recent advances in medical multi-modal models focus on specialized image analysis like dermatology, pathology, or radiology. However, they do not fully capture the complexity of real-world clinical diagnostics, which involve heterogeneous inputs and require ongoing contextual understanding during patient-physician interactions. To bridge this gap, we introduce PulseMind, a new family of multi-modal diagnostic models that integrates a systematically curated dataset, a comprehensive evaluation benchmark, and a tailored training framework. Specifically, we first construct a diagnostic dataset, MediScope, which comprises 98,000 real-world multi-turn consultations and 601,500 medical images, spanning over 10 major clinical departments and more than 200 sub-specialties. Then, to better reflect the requirements of real-world clinical diagnosis, we develop the PulseMind Benchmark, a multi-turn diagnostic consultation benchmark with a four-dimensional evaluation protocol comprising proactiveness, accuracy, usefulness, and language quality. Finally, we design a training framework tailored for multi-modal clinical diagnostics, centered around a core component named Comparison-based Reinforcement Policy Optimization (CRPO). Compared to absolute score rewards, CRPO uses relative preference signals from multi-dimensional com-parisons to provide stable and human-aligned training guidance. Extensive experiments demonstrate that PulseMind achieves competitive performance on both the diagnostic consultation benchmark and public medical benchmarks.

PulseMind: A Multi-Modal Medical Model for Real-World Clinical Diagnosis

TL;DR

PulseMind tackles the gap in real-world clinical diagnostics by integrating a large-scale multi-turn, multi-modal dataset (MediScope), a dedicated diagnostic benchmark (PulseMind Benchmark), and a relative-reward reinforcement learning framework (CRPO). It demonstrates strong performance on both the PulseMind Benchmark and 11 public medical QA benchmarks, indicating robust generalization across modalities and tasks. The approach emphasizes realistic data collection, multi-turn context, and human-aligned optimization, offering a practical foundation for clinical diagnostic dialogue AI. These advances have potential to improve diagnostic efficiency and consistency in real-world clinical settings where heterogeneous inputs and physician-patient interactions are common.

Abstract

Recent advances in medical multi-modal models focus on specialized image analysis like dermatology, pathology, or radiology. However, they do not fully capture the complexity of real-world clinical diagnostics, which involve heterogeneous inputs and require ongoing contextual understanding during patient-physician interactions. To bridge this gap, we introduce PulseMind, a new family of multi-modal diagnostic models that integrates a systematically curated dataset, a comprehensive evaluation benchmark, and a tailored training framework. Specifically, we first construct a diagnostic dataset, MediScope, which comprises 98,000 real-world multi-turn consultations and 601,500 medical images, spanning over 10 major clinical departments and more than 200 sub-specialties. Then, to better reflect the requirements of real-world clinical diagnosis, we develop the PulseMind Benchmark, a multi-turn diagnostic consultation benchmark with a four-dimensional evaluation protocol comprising proactiveness, accuracy, usefulness, and language quality. Finally, we design a training framework tailored for multi-modal clinical diagnostics, centered around a core component named Comparison-based Reinforcement Policy Optimization (CRPO). Compared to absolute score rewards, CRPO uses relative preference signals from multi-dimensional com-parisons to provide stable and human-aligned training guidance. Extensive experiments demonstrate that PulseMind achieves competitive performance on both the diagnostic consultation benchmark and public medical benchmarks.
Paper Structure (22 sections, 2 equations, 5 figures, 4 tables)

This paper contains 22 sections, 2 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Visual Language Models
  4. Medical Visual Language Models
  5. PulseMind
  6. Dataset
  7. MediScope.
  8. Public Datasets.
  9. Evaluation Benchmark
  10. PulseMind Benchmark
  11. General Medical QA Benchmarks.
  12. Training Framework
  13. Supervised Fine-Tuning.
  14. Reinforcement Learning with CRPO.
  15. Experiments
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Comparison between (a) conventional multi-modal medical VQA tasks and (b) real-world diagnostic dialogue scenarios.
  • Figure 2: Overview of PulseMind, including dataset construction (MediScope), the PulseMind Benchmark, and CRPO training."CP" denotes the counterpart model.
  • Figure 3: Characteristics of the collected multi-turn dialogue training set from four perspectives: (a) Distribution of major departments; (b) Heterogeneous Medical Image Modalities; (c) Distribution of multi-turn dialogue lengths; (d) Example of a multi-turn physician–patient consultation dialogue.
  • Figure 4: Win rates of our model against six baseline methods on the PulseMind Benchmark
  • Figure 5: Illustrative cases of six models on the PulseMind Benchmark. Representative response quality is color-coded, as indicated by the tags on the right.