Beyond N-grams: A Hierarchical Reward Learning Framework for Clinically-Aware Medical Report Generation

TL;DR

The paper addresses the unreliability of automatic medical report generation due to clinical hallucinations. It presents HiMed-RL, a hierarchical reward learning framework that guides generation through token-level fluency, concept-level factual grounding, and semantic-level diagnostic coherence, augmented by a Human-inspired Dynamic Reward Adjustment and an LLM verifier. Empirical results show state-of-the-art performance on multiple radiology datasets, with notable gains in out-of-domain generalization and semantic accuracy while maintaining a compact model size. This approach offers a robust pathway to clinically trustworthy MRG suitable for real-world deployment.

Abstract

Automatic medical report generation can greatly reduce the workload of doctors, but it is often unreliable for real-world deployment. Current methods can write formally fluent sentences but may be factually flawed, introducing serious medical errors known as clinical hallucinations, which make them untrustworthy for diagnosis. To bridge this gap, we introduce HiMed-RL, a Hierarchical Medical Reward Learning Framework designed to explicitly prioritize clinical quality. HiMed-RL moves beyond simple text matching by deconstructing reward learning into three synergistic levels: it first ensures linguistic fluency at the token-level, then enforces factual grounding at the concept-level by aligning key medical terms with expert knowledge, and finally assesses high-level diagnostic consistency at the semantic-level using a specialized LLM verifier. This hierarchical reward is implemented via a Human-inspired Dynamic Reward Adjustment, a strategy which first teaches the model to learn basic facts before progressing to more complex diagnostic reasoning. Experimentally, HiMed-3B achieves state-of-the-art performance on both in-domain and out-of-domain benchmarks, particularly on the latter, with an improvement of 12.1% over the second-best baseline. Our work provides a robust paradigm for generating reports that not only improve fluency but clinical fine-grained quality.

Figures (4)

• Figure 1: (a) Token-level, the report text is decomposed into individual words or sub-word tokens for assessment. Concept-level, relevant tokens are aggregated into clinically significant medical concepts, e.g., postsurgical changes. Semantic-level aims to comprehend the core meaning and diagnostic conclusion of the entire report. (b) Our HiMed-3B model outperforms Qwen2.5-VL-7B across three evaluation levels: at the token-level, with a comparable average BLEU score; at the concept-level, by correctly identifying three medical entity types and achieving superior ROUGE-L and METEOR scores; and at the semantic-level, demonstrating enhanced accuracy, relevance, and completeness.
• Figure 2: An overview of the HiMed-RL pipeline. MRG task is trained by a Hierarchical Reward Learning Framework that integrates token-level, concept-level, and semantic-level rewards. Human-inspired Dynamic Reward Adjustment strategy guides the model's learning process, transitioning from foundational fluency to complex medical reasoning, to optimize report quality.
• Figure 3: (a) Accuracy reward training curve. (b) Comparison of generated reports between the initial and transition phases.
• Figure 4: Ablation study results showcasing performance improvements in HiMed-3B across initial cold start and RL fine-tuning stages.