Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Models

TL;DR

Med-Evo is proposed, the first self-evolution framework for medical MLLMs that utilizes label-free reinforcement learning to promote model performance without requiring additional labeled data.

Abstract

Medical Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across diverse healthcare tasks. However, current post-training strategies, such as supervised fine-tuning and reinforcement learning, heavily depend on substantial annotated data while overlooking the potential of unlabeled test data for model enhancement. This limitation becomes particularly pronounced in medical domains, where acquiring extensive labeled medical data is difficult due to the strict data sensitivity and annotation complexity. Moreover, leveraging test data poses challenges in generating reliable supervision signals from unlabeled samples and maintaining stable self-evolution. To address these limitations, we propose Med-Evo, the first self-evolution framework for medical MLLMs that utilizes label-free reinforcement learning to promote model performance without requiring additional labeled data. Our framework introduces two key innovations: $1)$ Feature-driven Pseudo Labeling (FPL) that identifies semantic centroids from all heterogeneous candidate responses to select pseudo labels in each rollout, and $2)$ Hard-Soft Reward (HSR) that combines exact match with token-level assessment and semantic similarity to provide hierarchical reward. Experiments on three medical VQA benchmarks and two base MLLMs show clear advantages of our approach over SOTA methods, with significant improvements of 10.43\% accuracy and 4.68\% recall on the SLAKE dataset using Qwen2.5-VL, showing the effectiveness of our method.

Figures (3)

• Figure 1: Overview of medical MLLM adaptation strategies. (a) Direct inference leads to poor generalization on down-stream tasks; (b) Supervised learning requires extensive labeled data; (c) Our test-time self-evolution strategy using unlabeled data.
• Figure 2: Overview of our proposed Med-Evo framework for test-time self-evolution with four stages: $1)$ we prepare unlabeled data and the policy model $\pi_{\theta_{old}}$; $2)$ we employ Feature-driven Pseudo Labeling (FPL) to generate reliable supervision signals from unlabeled data (Sec. \ref{['FPL']}); $3)$ we utilize the Hard-soft Reward (HSR) mechanism to provide hierarchical reward signals by incorporating finer-grained evaluation strategies (Sec. \ref{['SHR']}); $4)$ we leverage Group Relative Policy Optimization (GRPO) to achieve iterative unsupervised self-evolution through relative advantage computation (Sec. \ref{['GRPO']}).
• Figure 3: Illustrative experiments based on Qwen2.5-VL-3B-Instruct on SLAKE dataset. (a) ablation study; (b) averaged reward score and model performance during the whole evolution process; (c) the hit rate comparison using base model.