MentalSeek-Dx: Towards Progressive Hypothetico-Deductive Reasoning for Real-world Psychiatric Diagnosis

TL;DR

The paper tackles the mismatch between clinical psychiatric decision-making and LLM-driven diagnosis by introducing MentalDx Bench, the first ecologically valid benchmark for disorder-level psychiatric diagnosis using real-world ICD-11 annotated EHRs. It identifies a paradigm misalignment where existing models excel at coarse category predictions but falter on fine-grained disorders, and proposes MentalSeek-Dx, a domain-specific model trained through Hypothetico-Deductive Trajectory Building and Process Reward-Based Curriculum Reinforcement Learning to internalize clinical reasoning. MentalSeek-Dx achieves state-of-the-art performance with 14B parameters, demonstrating that cognitive alignment with clinical decision processes, not just model size, is key to reliable psychiatric diagnosis. The work also provides a comprehensive knowledge-base, refinement pipeline, and evaluation framework, highlighting the importance of structured reasoning and human-in-the-loop safeguards in deploying AI-assisted psychiatric diagnostics.

Abstract

Mental health disorders represent a burgeoning global public health challenge. While Large Language Models (LLMs) have demonstrated potential in psychiatric assessment, their clinical utility is severely constrained by benchmarks that lack ecological validity and fine-grained diagnostic supervision. To bridge this gap, we introduce \textbf{MentalDx Bench}, the first benchmark dedicated to disorder-level psychiatric diagnosis within real-world clinical settings. Comprising 712 de-identified electronic health records annotated by board-certified psychiatrists under ICD-11 guidelines, the benchmark covers 76 disorders across 16 diagnostic categories. Evaluation of 18 LLMs reveals a critical \textit{paradigm misalignment}: strong performance at coarse diagnostic categorization contrasts with systematic failure at disorder-level diagnosis, underscoring a gap between pattern-based modeling and clinical hypothetico-deductive reasoning. In response, we propose \textbf{MentalSeek-Dx}, a medical-specialized LLM trained to internalize this clinical reasoning process through supervised trajectory construction and curriculum-based reinforcement learning. Experiments on MentalDx Bench demonstrate that MentalSeek-Dx achieves state-of-the-art (SOTA) performance with only 14B parameters, establishing a clinically grounded framework for reliable psychiatric diagnosis.

Figures (27)

• Figure 1: Diagnostic accuracy results on the MentalDx benchmark, evaluated across mainstream LLMs. The proposed model MentalSeek-Dx achieves state-of-the-art performance while operating at smaller scales.
• Figure 2: Overview of the MentalDx Benchmark, based on real-world medical centers, showing benchmark statistics, inputs, outputs, and data distributions.
• Figure 3: Statistics of expert-annotated errors across 360 cases. Results highlight a paradigm misalignment between LLM reasoning and clinical diagnostic logic.
• Figure 4: Overview of psychiatric knowledge base construction and clinically grounded corpus refinement.
• Figure 5: Training pipeline of MentalSeek-Dx to address paradigm misalignment by modeling clinically grounded diagnostic reasoning, consisting of two stages: 1) Hypothetico-Deductive Reasoning Trajectory Building using $\mathcal{K}$ as SFT corpus; 2) Reward-Based Curriculum Reinforcement Learning to guide the diagnostic process.