Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization

TL;DR

The paper tackles the challenge of authentically simulating depressive clients for clinical training, identifying biases in general-purpose LLMs that hinder realism. It presents Eeyore, an 8B model optimized via a three-stage alignment pipeline—Language-specific Alignment, Profile-Guided Role-Play, and Iterative Preference Optimization—to produce linguistically authentic, profile-consistent depression dialogues. By constructing real-world depression dialogue datasets, encoding structured psychological profiles, and applying a two-stage Direct Preference Optimization (with model-generated and expert preferences), Eeyore outperforms GPT-4o-based baselines in both linguistic authenticity and profile adherence. The work demonstrates the value of expert-in-the-loop alignment for clinically meaningful depression simulations and offers a pathway for deploying realistic role-play tools in mental health training.

Abstract

Large Language Models (LLMs) have been previously explored for mental healthcare training and therapy client simulation, but they still fall short in authentically capturing diverse client traits and psychological conditions. We introduce \textbf{Eeyore}, an 8B model optimized for realistic depression simulation through a structured alignment framework, incorporating expert input at every stage. First, we systematically curate real-world depression-related conversations, extracting depressive traits to guide data filtering and psychological profile construction, and use this dataset to instruction-tune Eeyore for profile adherence. Next, to further enhance realism, Eeyore undergoes iterative preference optimization -- first leveraging model-generated preferences and then calibrating with a small set of expert-annotated preferences. Throughout the entire pipeline, we actively collaborate with domain experts, developing interactive interfaces to validate trait extraction and iteratively refine structured psychological profiles for clinically meaningful role-play customization. Despite its smaller model size, the Eeyore depression simulation outperforms GPT-4o with SOTA prompting strategies, both in linguistic authenticity and profile adherence.

Figures (9)

• Figure 1: The alignment pipeline for optimizing LLMs to simulate individuals with depression in clinical training. Expert involvement is highlighted in green. Icons by kudinovs2024icon.
• Figure 2: Pipeline to input data for instruction-tuning.
• Figure 3: Overview of the two-stage Direct Preference Optimization process. 1 optimizes a DPO model from the instruction-tuned model using model-based preference data. 2 refines the DPO model with expert-annotated preferences, producing the final preference-optimized model.
• Figure 4: Expert evaluation scores comparing Eeyore with two baseline patient simulation approaches. Statistical comparisons were conducted using the Wilcoxon signed-rank test. $\ast$ indicates a statistically significant difference (p-value < 0.05). $\blacktriangle$ denotes a moderate effect size (0.3 - 0.5). $\blacktriangle\blacktriangle$ denotes a large effect size (>0.5), suggesting practical impact.
• Figure 5: Survey for evaluating psychological profile design. Experts reviewed each profile entry and suggested modifications or additional attributes to improve realism and relevance.