PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals

TL;DR

This work tackles the gap between CBT training and real-patient practice by introducing PATIENT-Ψ, a simulated patient framework that roots LLM-generated interactions in CBT-based cognitive models (CCD). It couples PATIENT-Ψ with PATIENT-Ψ-TRAINER to give trainees a structured, feedback-rich environment for formulating a patient’s cognitive model, supported by the 106-model Patient-Ψ-CM dataset and six conversational styles. In user studies with 20 experts and 13 trainees, PATIENT-Ψ demonstrated higher fidelity to real patients and PATIENT-Ψ-TRAINER outperformed GPT-4 baselines and traditional methods in perceived training effectiveness and confidence gains, though automatic evaluators showed some misalignment with human judgments. The approach promises scalable, interactive, and theory-grounded CBT training and may generalize to other therapeutic modalities, with future work including objective skill assessments and broader model comparisons.”

Abstract

Mental illness remains one of the most critical public health issues. Despite its importance, many mental health professionals highlight a disconnect between their training and actual real-world patient practice. To help bridge this gap, we propose PATIENT-Ψ, a novel patient simulation framework for cognitive behavior therapy (CBT) training. To build PATIENT-Ψ, we construct diverse patient cognitive models based on CBT principles and use large language models (LLMs) programmed with these cognitive models to act as a simulated therapy patient. We propose an interactive training scheme, PATIENT-Ψ-TRAINER, for mental health trainees to practice a key skill in CBT -- formulating the cognitive model of the patient -- through role-playing a therapy session with PATIENT-Ψ. To evaluate PATIENT-Ψ, we conducted a comprehensive user study of 13 mental health trainees and 20 experts. The results demonstrate that practice using PATIENT-Ψ-TRAINER enhances the perceived skill acquisition and confidence of the trainees beyond existing forms of training such as textbooks, videos, and role-play with non-patients. Based on the experts' perceptions, PATIENT-Ψ is perceived to be closer to real patient interactions than GPT-4, and PATIENT-Ψ-TRAINER holds strong promise to improve trainee competencies. Our code and data are released at \url{https://github.com/ruiyiw/patient-psi}.

Figures (27)

• Figure 1: Illustration of our patient simulation idea.
• Figure 2: The overall framework of Patient-$\Psi$ and Patient-$\Psi$-Trainer. We integrate the expert-created cognitive model with GPT-4 to build Patient-$\Psi$. In Patient-$\Psi$-Trainer, the trainee role-plays a therapy session with Patient-$\Psi$ to formulate its cognitive model. The trainee can compare their formulation with the cognitive model used to build Patient-$\Psi$ to get feedback.
• Figure 3: Fidelity of Patient-$\Psi$ and training effectiveness of Patient-$\Psi$-Trainer compared to GPT-4 baseline along multiple dimensions. X-axis: the % of experts who voted for a specific option; y-axis: the assessment dimension. Malad. means maladaptive, Think. means thinking, and Ident. means identification. Patient-$\Psi$ more closely resembles real patients (fidelity, left) and is considered more effective for trainees (training effectiveness, right).
• Figure 4: Experts rate 97% of the Patient-$\Psi$ patients as at least moderately accurate in reflecting the reference cognitive model. Intermed. means Intermediate.
• Figure 5: Mean overall fidelity of Patient-$\Psi$ and baseline as evaluated by experts and LLMs. Compared to experts, both GPT-4 and Llama 3 demonstrate opposite trends.