Predicting LLM Correctness in Prosthodontics Using Metadata and Hallucination Signals

TL;DR

This paper tackles the challenge of predicting LLM correctness in high-stakes medical education by leveraging metadata and hallucination signals. It evaluates GPT-4o and OSS-120B on a 98-question prosthodontics exam under three prompting strategies, using LOOCV for robustness. The key finding is that metadata alone can boost correctness prediction by up to $+7.14\%$ with a peak precision of $83.12\%$, while actual hallucination signals offer strong indicators only in idealized oracle conditions; prompting strategies influence the predictive value of metadata more than overall accuracy. The work demonstrates a promising direction for reliability signals in LLMs but also underscores that current methods are not robust enough for critical, high-stakes deployment in medical education, necessitating cautious, supervised use and further domain-specific research.

Abstract

Large language models (LLMs) are increasingly adopted in high-stakes domains such as healthcare and medical education, where the risk of generating factually incorrect (i.e., hallucinated) information is a major concern. While significant efforts have been made to detect and mitigate such hallucinations, predicting whether an LLM's response is correct remains a critical yet underexplored problem. This study investigates the feasibility of predicting correctness by analyzing a general-purpose model (GPT-4o) and a reasoning-centric model (OSS-120B) on a multiple-choice prosthodontics exam. We utilize metadata and hallucination signals across three distinct prompting strategies to build a correctness predictor for each (model, prompting) pair. Our findings demonstrate that this metadata-based approach can improve accuracy by up to +7.14% and achieve a precision of 83.12% over a baseline that assumes all answers are correct. We further show that while actual hallucination is a strong indicator of incorrectness, metadata signals alone are not reliable predictors of hallucination. Finally, we reveal that prompting strategies, despite not affecting overall accuracy, significantly alter the models' internal behaviors and the predictive utility of their metadata. These results present a promising direction for developing reliability signals in LLMs but also highlight that the methods explored in this paper are not yet robust enough for critical, high-stakes deployment.

Figures (8)

• Figure 1: Dataset Overview. First row in grey denotes Primary Data Field. Second row in blue denotes Inference Data Field. Third row in cyan denotes LLM-as-a-Judge Data Field.
• Figure 2: Different prompting strategies have a non-significant effect on model accuracy.
• Figure 3: Different prompting strategies have a non-significant effect on model consistency.
• Figure 4: Despite similar performance, each pair of prompting strategies shows some degree of disagreement.
• Figure 5: Correlation of each metadata and hallucination rate with respect to each model's correctness. We group safe and benign extrinsic hallucinations into one group (non-harmful extrinsic hallucination) and compare it to harmful extrinsic hallucination.