BioACE: An Automated Framework for Biomedical Answer and Citation Evaluations

TL;DR

BioACE presents an automated framework for evaluating biomedical answers and citations produced by AI systems, addressing the critical need for factual and evidence-grounded biomedical QA. It operationalizes answer evaluation along four axes—completeness, correctness, precision, and recall—via nugget-based matching, PLM/LLM-based classification, and similarity-based and NLI-driven evidence assessment, while evaluating citation support through three task settings. The framework is validated on MedAESQA and the BioGen/BioGen2024 data, identifying robust PLM-based approaches for answer correctness and nugget matching, while highlighting challenges in citation attribution and the variability of LLM-based methods. BioACE provides a practical, open-source evaluation package for automated biomedical answer and citation assessment, offering a reusable tool to improve reliability and transparency in AI-assisted biomedical information provision.

Abstract

With the increasing use of large language models (LLMs) for generating answers to biomedical questions, it is crucial to evaluate the quality of the generated answers and the references provided to support the facts in the generated answers. Evaluation of text generated by LLMs remains a challenge for question answering, retrieval-augmented generation (RAG), summarization, and many other natural language processing tasks in the biomedical domain, due to the requirements of expert assessment to verify consistency with the scientific literature and complex medical terminology. In this work, we propose BioACE, an automated framework for evaluating biomedical answers and citations against the facts stated in the answers. The proposed BioACE framework considers multiple aspects, including completeness, correctness, precision, and recall, in relation to the ground-truth nuggets for answer evaluation. We developed automated approaches to evaluate each of the aforementioned aspects and performed extensive experiments to assess and analyze their correlation with human evaluations. In addition, we considered multiple existing approaches, such as natural language inference (NLI) and pre-trained language models and LLMs, to evaluate the quality of evidence provided to support the generated answers in the form of citations into biomedical literature. With the detailed experiments and analysis, we provide the best approaches for biomedical answer and citation evaluation as a part of BioACE (https://github.com/deepaknlp/BioACE) evaluation package.

Figures (7)

• Figure 1: Answer correctness performance comparison of the embedding models, their derived cosine similarity, and the NLI model on multiple evaluation metrics.
• Figure 2: Prompt used for answer completeness evaluation.
• Figure 3: Prompt used for binary labeling of citations.
• Figure 4: Prompt used for ternary labeling of citations.
• Figure 5: Prompt used for labeling answer and document nugget lists.