Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solving

TL;DR

Problem: assessing whether advances in LLM reasoning yield flexible clinical problem-solving beyond rote pattern matching. Approach: evaluate reasoning-enabled models from multiple vendors on mARC, an adversarial medical QA benchmark designed to provoke the Einstellung bias, using chain-of-thought prompts and uncertainty calibration. Findings: stronger reasoning models show reduced susceptibility to the Einstellung effect, approaching human performance on mARC and outperforming humans on the most challenging items; uncertainty estimates improve calibration. Significance: indicates potential for clinician-augmented decision support with deferral-based strategies, while highlighting the need for broader validation and safeguards for real-world deployment.

Abstract

Large Language Models (LLMs) have achieved high accuracy on medical question-answer (QA) benchmarks, yet their capacity for flexible clinical reasoning has been debated. Here, we asked whether advances in reasoning LLMs improve their cognitive flexibility in clinical reasoning. We assessed reasoning models from the OpenAI, Grok, Gemini, Claude, and DeepSeek families on the medicine abstraction and reasoning corpus (mARC), an adversarial medical QA benchmark which utilizes the Einstellung effect to induce inflexible overreliance on learned heuristic patterns in contexts where they become suboptimal. We found that strong reasoning models avoided Einstellung-based traps more often than weaker reasoning models, achieving human-level performance on mARC. On questions most commonly missed by physicians, the top 5 performing models answered 55% to 70% correctly with high confidence, indicating that these models may be less susceptible than humans to Einstellung effects. Our results indicate that strong reasoning models demonstrate improved flexibility in medical reasoning, achieving performance on par with humans on mARC.

Figures (6)

• Figure 1: Demonstration of Einstellung effect evoked by failure to override default heuristics. The vignette presents a cue ($C$) that triggers a default heuristic for brain bleed ($H$), and a blocker ($B$) that, together with background medical knowledge $K$ (e.g., that intracranial hemorrhage requires a brain and anencephaly entails absence of brain), entails $\neg H$. An Einstellung effect type reasoning failure occurs when the model prioritizes the default heuristic $C \Rightarrow_{\text{def}} H$ over the deductive constraint $K \cup \{B\} \vdash \neg H$.
• Figure 2: Comparison of model performance on mARC. Colored bars indicate model accuracies with 95% bootstrap confidence intervals (CI) denoted by black range bar. The rightmost bar shows average human performance across 5 physicians with mean 0.66 and 95% CI 0.55 to 0.75. Accuracies shown for LLMs are mean accuracies across 15 runs. Advanced reasoning models (hatched bars) demonstrated improved performanced compared to their non-reasoning counterparts (solid bars). No significant difference in performance was observed between the Claude 4.1 Opus, Gemini 2.5-Pro, GPT-5.1, and Grok-4-Fast-Reasoning and human performance (paired bootstrap test). The best performing model was Claude 4.1 Opus with mean performance of 0.75 [95% CI of 0.74-0.76].
• Figure 3: A newer reasoning model rejects the stereotyped “recheck BP” reflex at an invalid site, recognizes unreliable measurement, and treats the patient based on clinical status. This reflects information sufficiency and context re‑anchoring rather than rote, inflexible pattern completion.
• Figure 4: The strong reasoning model defeats a high‑frequency anticoagulation $\rightarrow$ hemorrhage lure by applying strict logical negation.
• Figure 5: Uncertainty estimation on mARC using entropy‑ and agreement‑based sample‑consistency (SC). Newer/larger reasoning models showed improvements in accuracy, Brier scores and supplied usable confidence signals. Compared with baseline performance on MMLU‑Pro (black markers), mARC induces a distribution shift; nevertheless, SC remains informative for deferral.