Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis

Wenkai Zhao, Zipei Wang, Mengjie Fang, Di Dong, Jie Tian, Lingwei Zhang

Abstract

General Multimodal Large Language Models (MLLMs) often underperform in capturing domain-specific nuances in medical diagnosis, trailing behind fully supervised baselines. Although fine-tuning provides a remedy, the high costs of expert annotation and massive computational overhead limit its scalability. To bridge this gap without updating the weights of the pre-trained backbone of the MLLM, we propose a Clinician Mimetic Workflow. This is a novel In-Context Learning (ICL) framework designed to synergize Discriminative Exemplar Coreset Selection (DECS) and Self-Refined Experience Summarization (SRES). Specifically, DECS simulates a clinician's ability to reference "anchor cases" by selecting discriminative visual coresets from noisy data at the computational level; meanwhile, SRES mimics the cognition and reflection in clinical diagnosis by distilling diverse rollouts into a dynamic textual Experience Bank. Extensive evaluation across all 12 datasets of the MedMNIST 2D benchmark demonstrates that our method outperforms zero-shot general and medical MLLMs. Simultaneously, it achieves performance levels comparable to fully supervised vision models and domain-specific fine-tuned MLLMs, setting a new benchmark for parameter-efficient medical in-context learning. Our code is available at an anonymous repository: https://anonymous.4open.science/r/Synergizing-Discriminative-Exemplars-and-Self-Refined-Experience-ED74.

Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis

Abstract

General Multimodal Large Language Models (MLLMs) often underperform in capturing domain-specific nuances in medical diagnosis, trailing behind fully supervised baselines. Although fine-tuning provides a remedy, the high costs of expert annotation and massive computational overhead limit its scalability. To bridge this gap without updating the weights of the pre-trained backbone of the MLLM, we propose a Clinician Mimetic Workflow. This is a novel In-Context Learning (ICL) framework designed to synergize Discriminative Exemplar Coreset Selection (DECS) and Self-Refined Experience Summarization (SRES). Specifically, DECS simulates a clinician's ability to reference "anchor cases" by selecting discriminative visual coresets from noisy data at the computational level; meanwhile, SRES mimics the cognition and reflection in clinical diagnosis by distilling diverse rollouts into a dynamic textual Experience Bank. Extensive evaluation across all 12 datasets of the MedMNIST 2D benchmark demonstrates that our method outperforms zero-shot general and medical MLLMs. Simultaneously, it achieves performance levels comparable to fully supervised vision models and domain-specific fine-tuned MLLMs, setting a new benchmark for parameter-efficient medical in-context learning. Our code is available at an anonymous repository: https://anonymous.4open.science/r/Synergizing-Discriminative-Exemplars-and-Self-Refined-Experience-ED74.

Paper Structure

This paper contains 19 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Overall Architecture
  4. Discriminative Exemplar Coreset Selection
  5. Feature Extracting and Adaptive Initialization
  6. Iterative Optimization of Distinctive Cases
  7. Inference-time Exemplar Referencing
  8. Self-Refined Experience Summarization
  9. Contextual Formulation
  10. Reward-Guided Rollout and Contrastive Summarization
  11. Experience Bank Update
  12. Experiments and Results
  13. Experimental Settings
  14. Datasets
  15. Implementation Details
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Overview of the proposed Clinician Mimetic Workflow. (a) The diagnostic cognitive process of human clinicians. (b) Discriminative Exemplar Coreset Selection, where data points of the same color belong to the same class. (c) Self-Refined Experience Summarization, dynamically updating the Experience Bank via reward-guided rollouts.
  • Figure 2: Illustration of the Self-Refined Experience Summarization. Texts highlighted in identical colors map the critical semantic discrepancies between distinct rollouts directly to the resulting synthesized Diagnostic Heuristics.