Evaluating Large Language Models for Fair and Reliable Organ Allocation
Brian Hyeongseok Kim, Hannah Murray, Isabelle Lee, Jason Byun, Joshua Lum, Dani Yogatama, Evi Micha
TL;DR
This work addresses the challenge of using large language models for organ allocation by evaluating fairness and decision reliability with real-world OPTN kidney data. It introduces two allocation tasks, Choose-One and Rank-All, and adapts fairness metrics from ranking theory ($rND$ and exposure) to assess demographic bias in both selection and ranking contexts. The study finds substantial model- and task-dependent disparities: while exposure-based fairness often suggests equitable outcomes, probability-based metrics reveal systematic, group-specific sorting biases, especially at top ranks; results also show that decision context can invert observed biases across models. The findings highlight the need for rigorous, rank-aware fairness evaluation and human oversight before adopting LLMs in high-stakes organ allocation, and establish a methodological foundation for auditing LLM-driven clinical decisions.
Abstract
Medical institutions are considering the use of LLMs in high-stakes clinical decision-making, such as organ allocation. In such sensitive use cases, evaluating fairness is imperative. However, existing evaluation methods often fall short; benchmarks are too simplistic to capture real-world complexity, and accuracy-based metrics fail to address the absence of a clear ground truth. To realistically and fairly model organ allocation, specifically kidney allocation, we begin by testing the medical knowledge of LLMs to determine whether they understand the clinical factors required to make sound allocation decisions. Building on this foundation, we design two tasks: (1) Choose-One and (2) Rank-All. In Choose-One, LLMs select a single candidate from a list of potential candidates to receive a kidney. In this scenario, we assess fairness across demographics using traditional fairness metrics, such as proportional parity. In Rank-All, LLMs rank all candidates waiting for a kidney, reflecting real-world allocation processes more closely, where an organ is passed down a ranked list until allocated. Our evaluation on three LLMs reveals a divergence between fairness metrics: while exposure-based metrics suggest equitable outcomes, probability-based metrics uncover systematic preferential sorting, where specific groups were clustered in upper-ranking tiers. Furthermore, we observe that demographic preferences are highly task-dependent, showing inverted trends between Choose-One and Rank-All tasks, even when considering the topmost rank. Overall, our results indicate that current LLMs can introduce inequalities in real-world allocation scenarios, underscoring the urgent need for rigorous fairness evaluation and human oversight before their use in high-stakes decision-making.