Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosis

TL;DR

The paper investigates how large language models perform in diagnosing Rheumatoid Arthritis (RA) using real-world patient data, revealing a critical misalignment: models can predict RA with high accuracy while generating flawed or clinically unsound explanations. It introduces the PreRAID dataset and a retrieval-augmented framework to evaluate multiple LLMs across various knowledge-base sizes and conversational architectures, including single and multi-agent setups. Although diagnostic accuracy often exceeds 90% (with some configurations approaching 100%), expert reviews show that the reasoning behind these predictions is frequently incorrect, exposing a gap between performance and interpretability. The findings highlight the importance of robust, medically valid explanations for clinical use and call for advances in prompts, reasoning verification, and explanatory mechanisms to ensure safe deployment of LLM-based pre-screening tools in healthcare.

Abstract

Large language models (LLMs) offer a promising pre-screening tool, improving early disease detection and providing enhanced healthcare access for underprivileged communities. The early diagnosis of various diseases continues to be a significant challenge in healthcare, primarily due to the nonspecific nature of early symptoms, the shortage of expert medical practitioners, and the need for prolonged clinical evaluations, all of which can delay treatment and adversely affect patient outcomes. With impressive accuracy in prediction across a range of diseases, LLMs have the potential to revolutionize clinical pre-screening and decision-making for various medical conditions. In this work, we study the diagnostic capability of LLMs for Rheumatoid Arthritis (RA) with real world patients data. Patient data was collected alongside diagnoses from medical experts, and the performance of LLMs was evaluated in comparison to expert diagnoses for RA disease prediction. We notice an interesting pattern in disease diagnosis and find an unexpected \textit{misalignment between prediction and explanation}. We conduct a series of multi-round analyses using different LLM agents. The best-performing model accurately predicts rheumatoid arthritis (RA) diseases approximately 95\% of the time. However, when medical experts evaluated the reasoning generated by the model, they found that nearly 68\% of the reasoning was incorrect. This study highlights a clear misalignment between LLMs high prediction accuracy and its flawed reasoning, raising important questions about relying on LLM explanations in clinical settings. \textbf{LLMs provide incorrect reasoning to arrive at the correct answer for RA disease diagnosis.}

Figures (9)

• Figure 1: Diagram shown to the patient for indicating pain locations.
• Figure 2: Overview of the framework for RA patients diagnosis. We conduct experiments with four different architectures: LLM without knowledge base (from PreRAID), LLM+knowledge base+RAG, 2 LLM Agents+knowledge base+RAG, 3 LLM Agents+knowledge base+RAG. We store the final diagnosis and the reasoning tokens generated by each architecture.
• Figure 3: Accuracy of disease prediction across all the models by providing all the knowledge bases. Orange, sky blue, deep blue and light blue color represents the accuracy in llm without RAG, single agent+RAG, 2 agents+RAG, and 3 agents+RAG, respectively.
• Figure 4: Comparison of diagnostic accuracy and expert-rated reasoning ratings for Gemini 2.0 Flash and GPT-4o across four architectures: LLM without RAG, LLM+RAG, 2 LLM agents+RAG, and 3 LLM agents+RAG
• Figure 5: Reasoning performance across different LLM models in the 2 LLM agents+RAG setup. This figure compares the reasoning quality of various LLM models: GPT-3.5 Turbo, GPT-4o, GPT-4o-mini, Gemini 1.5 Flash, Gemini 2.0 Flash, and QWEN-2-7B, within the 2 LLM agents+RAG framework.