Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs

TL;DR

This paper addresses a gap in clinical AI evaluation by arguing that traditional QA benchmarks (e.g., MedQA) fail to capture real-world clinical decision-making. It introduces a two-dimensional paradigm based on Clinical Backgrounds and Clinical Questions to classify tasks, summarizes existing datasets along these axes, and reviews training-time and test-time techniques (SFT, RL, CoT, RAG, multi-agent systems) for improving decision-making. It also extends evaluation beyond accuracy to efficiency and explainability, proposing metrics and LLM-based judging approaches, and identifies open issues such as realistic data collection, redundancy filtering, and multi-agent evaluation. The work aims to standardize task definitions and benchmarking to guide the development of clinically meaningful LLMs that perform robustly in real-world settings and workflows.

Abstract

Large language models (LLMs) show promise for clinical use. They are often evaluated using datasets such as MedQA. However, Many medical datasets, such as MedQA, rely on simplified Question-Answering (Q\A) that underrepresents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing datasets and benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs.

Figures (3)

• Figure 1: Three different clinical settings (No clinical background setting is not shown in the figure).
• Figure 2: Four different clinical question settings.
• Figure 3: The clinical background and clinical question settings of mainstream medical datasets/benchmarks. Note that some datasets/benchmarks may have multiple settings.