Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation

Xiechi Zhang, Zetian Ouyang, Linlin Wang, Gerard de Melo, Zhu Cao, Xiaoling Wang, Ya Zhang, Yanfeng Wang, Liang He

TL;DR

This work presents AutoMedEval, an open-source 13B evaluator designed to automatically assess the QA capabilities of medical LLMs with high fidelity to human judgments. It introduces a retrieval-augmented instruction dataset and a hierarchical training regime comprising curriculum instruction tuning and iterative knowledge introspection to embed professional medical evaluation criteria. Extensive human evaluations and ablation studies show AutoMedEval outperforming baselines and achieving strong correlations with expert judgments, while remaining open and reproducible for privacy-conscious medical settings. The method offers a data-efficient path to reliable, scalable medical model evaluation with potential to reduce reliance on costly human assessment.

Abstract

With the proliferation of large language models (LLMs) in the medical domain, there is increasing demand for improved evaluation techniques to assess their capabilities. However, traditional metrics like F1 and ROUGE, which rely on token overlaps to measure quality, significantly overlook the importance of medical terminology. While human evaluation tends to be more reliable, it can be very costly and may as well suffer from inaccuracies due to limits in human expertise and motivation. Although there are some evaluation methods based on LLMs, their usability in the medical field is limited due to their proprietary nature or lack of expertise. To tackle these challenges, we present AutoMedEval, an open-sourced automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs. The overarching objective of AutoMedEval is to assess the quality of responses produced by diverse models, aspiring to significantly reduce the dependence on human evaluation. Specifically, we propose a hierarchical training method involving curriculum instruction tuning and an iterative knowledge introspection mechanism, enabling AutoMedEval to acquire professional medical assessment capabilities with limited instructional data. Human evaluations indicate that AutoMedEval surpasses other baselines in terms of correlation with human judgments.

AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation

TL;DR

This work presents AutoMedEval, an open-source 13B evaluator designed to automatically assess the QA capabilities of medical LLMs with high fidelity to human judgments. It introduces a retrieval-augmented instruction dataset and a hierarchical training regime comprising curriculum instruction tuning and iterative knowledge introspection to embed professional medical evaluation criteria. Extensive human evaluations and ablation studies show AutoMedEval outperforming baselines and achieving strong correlations with expert judgments, while remaining open and reproducible for privacy-conscious medical settings. The method offers a data-efficient path to reliable, scalable medical model evaluation with potential to reduce reliance on costly human assessment.

Abstract

With the proliferation of large language models (LLMs) in the medical domain, there is increasing demand for improved evaluation techniques to assess their capabilities. However, traditional metrics like F1 and ROUGE, which rely on token overlaps to measure quality, significantly overlook the importance of medical terminology. While human evaluation tends to be more reliable, it can be very costly and may as well suffer from inaccuracies due to limits in human expertise and motivation. Although there are some evaluation methods based on LLMs, their usability in the medical field is limited due to their proprietary nature or lack of expertise. To tackle these challenges, we present AutoMedEval, an open-sourced automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs. The overarching objective of AutoMedEval is to assess the quality of responses produced by diverse models, aspiring to significantly reduce the dependence on human evaluation. Specifically, we propose a hierarchical training method involving curriculum instruction tuning and an iterative knowledge introspection mechanism, enabling AutoMedEval to acquire professional medical assessment capabilities with limited instructional data. Human evaluations indicate that AutoMedEval surpasses other baselines in terms of correlation with human judgments.
Paper Structure (39 sections, 7 equations, 9 figures, 8 tables, 2 algorithms)

This paper contains 39 sections, 7 equations, 9 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Task Formulation
  3. Methodology
  4. Overview
  5. Retrieval Augmented Instruction Dataset
  6. Reliablity Verification
  7. Curriculum Instruction Tuning
  8. Iterative Knowledge Introspection
  9. Experiments
  10. Baselines and Test Set
  11. Human Annotation and Judgement
  12. Evaluation Methods
  13. Score Evaluation
  14. Rationale Evaluation
  15. Double-blind Preference
  16. ...and 24 more sections

Figures (9)

  • Figure 1: Typical example of medical LLMs' responses evaluation.
  • Figure 2: Creation of our instruction dataset and automatic evaluation model AutoMedEval.
  • Figure 3: Collaborative Knowledge Introspection
  • Figure 4: AutoMedEval and chief physicians' judgment on every two medical LLMs' performance. "Win" means the ratio of cases where the current medical LLM outperforms another, while "Tie" means both medical LLMs' scores are the same.
  • Figure 5: (a) Human assessment results on AutoMedEval's evaluation content. Know. Cor. represents Knowledge Correctness and Ref. Cor. means Reference Correctness. (b) Double-blind preference experiment results.
  • ...and 4 more figures