Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients

Armin Berger, Manuela Bergau, Helen Schneider, Saad Ahmad, Tom Anglim Lagones, Gianluca Brugnara, Martha Foltyn-Dumitru, Kai Schlamp, Philipp Vollmuth, Rafet Sifa

TL;DR

The paper investigates whether R1-style training—combining supervised fine-tuning with policy optimization—improves small medical vision-language models for multilabel chest X-ray classification under extreme resource constraints. ChexReason demonstrates that GRPO can raise CheXpert benchmark performance but weaken cross-dataset transfer to NIH, while SFT can improve NIH generalization, revealing a generalization paradox tied to benchmark-driven optimization. A key finding is that the effectiveness of instruction formats depends on medical pre-training: medically pre-trained models benefit from direct label outputs, whereas general-purpose models require structured reasoning scaffolds to compensate for domain knowledge gaps. The work suggests that, for robust clinical deployment across diverse populations, carefully curated supervised fine-tuning may outperform aggressive RL optimization on hard benchmarks, especially in resource-limited settings.

Abstract

Recent Reinforcement Learning (RL) advances for Large Language Models (LLMs) have improved reasoning tasks, yet their resource-constrained application to medical imaging remains underexplored. We introduce ChexReason, a vision-language model trained via R1-style methodology (SFT followed by GRPO) using only 2,000 SFT samples, 1,000 RL samples, and a single A100 GPU. Evaluations on CheXpert and NIH benchmarks reveal a fundamental tension: GRPO recovers in-distribution performance (23% improvement on CheXpert, macro-F1 = 0.346) but degrades cross-dataset transferability (19% drop on NIH). This mirrors high-resource models like NV-Reason-CXR-3B, suggesting the issue stems from the RL paradigm rather than scale. We identify a generalization paradox where the SFT checkpoint uniquely improves on NIH before optimization, indicating teacher-guided reasoning captures more institution-agnostic features. Furthermore, cross-model comparisons show structured reasoning scaffolds benefit general-purpose VLMs but offer minimal gain for medically pre-trained models. Consequently, curated supervised fine-tuning may outperform aggressive RL for clinical deployment requiring robustness across diverse populations.

Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients

TL;DR

The paper investigates whether R1-style training—combining supervised fine-tuning with policy optimization—improves small medical vision-language models for multilabel chest X-ray classification under extreme resource constraints. ChexReason demonstrates that GRPO can raise CheXpert benchmark performance but weaken cross-dataset transfer to NIH, while SFT can improve NIH generalization, revealing a generalization paradox tied to benchmark-driven optimization. A key finding is that the effectiveness of instruction formats depends on medical pre-training: medically pre-trained models benefit from direct label outputs, whereas general-purpose models require structured reasoning scaffolds to compensate for domain knowledge gaps. The work suggests that, for robust clinical deployment across diverse populations, carefully curated supervised fine-tuning may outperform aggressive RL optimization on hard benchmarks, especially in resource-limited settings.

Abstract

Recent Reinforcement Learning (RL) advances for Large Language Models (LLMs) have improved reasoning tasks, yet their resource-constrained application to medical imaging remains underexplored. We introduce ChexReason, a vision-language model trained via R1-style methodology (SFT followed by GRPO) using only 2,000 SFT samples, 1,000 RL samples, and a single A100 GPU. Evaluations on CheXpert and NIH benchmarks reveal a fundamental tension: GRPO recovers in-distribution performance (23% improvement on CheXpert, macro-F1 = 0.346) but degrades cross-dataset transferability (19% drop on NIH). This mirrors high-resource models like NV-Reason-CXR-3B, suggesting the issue stems from the RL paradigm rather than scale. We identify a generalization paradox where the SFT checkpoint uniquely improves on NIH before optimization, indicating teacher-guided reasoning captures more institution-agnostic features. Furthermore, cross-model comparisons show structured reasoning scaffolds benefit general-purpose VLMs but offer minimal gain for medically pre-trained models. Consequently, curated supervised fine-tuning may outperform aggressive RL for clinical deployment requiring robustness across diverse populations.
Paper Structure (13 sections, 4 equations, 1 figure, 6 tables)

This paper contains 13 sections, 4 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dataset
  5. Training
  6. Supervised Fine-Tuning
  7. Group Relative Policy Optimization
  8. Results and Evaluation
  9. Effect of Prompting
  10. Effect of Supervised Fine-Tuning
  11. Effect of Group Relative Policy Optimization
  12. Generalization Analysis
  13. Conclusion

Figures (1)

  • Figure 1: Comparative training dynamics across all four supervised fine-tuning variants. The right panel shows training loss convergence, while the left panel displays mean token-level prediction accuracy. Notably, syntax-constrained variants (Only Label, Reasoning A, Reasoning Narrative) converge rapidly to higher token accuracy saturation, whereas Free Reasoning exhibits slower convergence and lower saturation levels. This pattern reflects fundamental differences in output entropy rather than learning quality, with Free Reasoning's diverse, unstructured traces requiring more nuanced semantic learning compared to the predictable template patterns of structured formats.