Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Accuracy: Risk-Sensitive Evaluation of Hallucinated Medical Advice

Savan Doshi

TL;DR

The paper tackles the problem that factual correctness alone is insufficient for evaluating hallucinations in patient-facing medical QA. It introduces a risk-sensitive evaluation framework that uses a Risk-Sensitive Hallucination Score (RSHS) to quantify risk-bearing language and a relevance metric (QASim) to assess grounding, enabling detection of high-risk, poorly grounded outputs. Empirically, it shows substantial variation in risk profiles across models with similar surface behavior and reveals strong dependence on prompts and task framing, which standard metrics miss. The results argue for incorporating risk sensitivity into evaluation to better align with clinical safety considerations, and they demonstrate that the framework generalizes from encoder–decoder to decoder-only architectures. Collectively, the approach provides a lightweight, interpretable diagnostic tool for probing safety-critical behavior in medical LLMs and guiding safer prompt design.

Abstract

Large language models are increasingly being used in patient-facing medical question answering, where hallucinated outputs can vary widely in potential harm. However, existing hallucination standards and evaluation metrics focus primarily on factual correctness, treating all errors as equally severe. This obscures clinically relevant failure modes, particularly when models generate unsupported but actionable medical language. We propose a risk-sensitive evaluation framework that quantifies hallucinations through the presence of risk-bearing language, including treatment directives, contraindications, urgency cues, and mentions of high-risk medications. Rather than assessing clinical correctness, our approach evaluates the potential impact of hallucinated content if acted upon. We further combine risk scoring with a relevance measure to identify high-risk, low-grounding failures. We apply this framework to three instruction-tuned language models using controlled patient-facing prompts designed as safety stress tests. Our results show that models with similar surface-level behavior exhibit substantially different risk profiles and that standard evaluation metrics fail to capture these distinctions. These findings highlight the importance of incorporating risk sensitivity into hallucination evaluation and suggest that evaluation validity is critically dependent on task and prompt design.

Beyond Accuracy: Risk-Sensitive Evaluation of Hallucinated Medical Advice

TL;DR

The paper tackles the problem that factual correctness alone is insufficient for evaluating hallucinations in patient-facing medical QA. It introduces a risk-sensitive evaluation framework that uses a Risk-Sensitive Hallucination Score (RSHS) to quantify risk-bearing language and a relevance metric (QASim) to assess grounding, enabling detection of high-risk, poorly grounded outputs. Empirically, it shows substantial variation in risk profiles across models with similar surface behavior and reveals strong dependence on prompts and task framing, which standard metrics miss. The results argue for incorporating risk sensitivity into evaluation to better align with clinical safety considerations, and they demonstrate that the framework generalizes from encoder–decoder to decoder-only architectures. Collectively, the approach provides a lightweight, interpretable diagnostic tool for probing safety-critical behavior in medical LLMs and guiding safer prompt design.

Abstract

Large language models are increasingly being used in patient-facing medical question answering, where hallucinated outputs can vary widely in potential harm. However, existing hallucination standards and evaluation metrics focus primarily on factual correctness, treating all errors as equally severe. This obscures clinically relevant failure modes, particularly when models generate unsupported but actionable medical language. We propose a risk-sensitive evaluation framework that quantifies hallucinations through the presence of risk-bearing language, including treatment directives, contraindications, urgency cues, and mentions of high-risk medications. Rather than assessing clinical correctness, our approach evaluates the potential impact of hallucinated content if acted upon. We further combine risk scoring with a relevance measure to identify high-risk, low-grounding failures. We apply this framework to three instruction-tuned language models using controlled patient-facing prompts designed as safety stress tests. Our results show that models with similar surface-level behavior exhibit substantially different risk profiles and that standard evaluation metrics fail to capture these distinctions. These findings highlight the importance of incorporating risk sensitivity into hallucination evaluation and suggest that evaluation validity is critically dependent on task and prompt design.
Paper Structure (31 sections, 1 equation, 2 figures, 1 table)

This paper contains 31 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Risk-Sensitive Evaluation Framework
  3. From Hallucination Presence to Hallucination Risk
  4. Risk-Bearing Language Categories
  5. Risk-Sensitive Hallucination Score
  6. Risk × Relevance Analysis
  7. Scope and Intended Use
  8. Experimental Setup and Results
  9. Patient-Facing Prompt Design
  10. Models
  11. Evaluation Metrics
  12. Overall Risk Profiles
  13. Risk Category Analysis
  14. Risk--Relevance Failure Modes
  15. Prompt Sensitivity
  16. ...and 16 more sections

Figures (2)

  • Figure 1: Distribution of Risk-Sensitive Hallucination Scores (RSHS) across three instruction-tuned models on 200 patient-facing prompts. Higher values indicate more risk-bearing medical language. Differences are most pronounced in the upper tail, suggesting model-dependent variation in risk-bearing generations.
  • Figure 2: Risk–relevance analysis of model responses. Each point represents a patient-facing response, plotted by its Risk-Sensitive Hallucination Score (RSHS) and query–response relevance (QASim). While many high-risk responses remain semantically aligned with the input, a subset exhibits elevated risk and weak relevance, corresponding to unsupported or weakly grounded medical advice.