Adversarial Training for Failure-Sensitive User Simulation in Mental Health Dialogue Optimization

TL;DR

This work tackles the challenge of evaluating task-oriented dialogue systems with realistic user simulators in a mental health domain. It introduces an adversarial framework where a neural user simulator competes with a discriminator and is refined via Direct Preference Optimization, leveraging rich, hierarchical context to align with real user behavior. The approach yields substantial gains over zero-shot models in linguistic and behavioral realism, increases diversity, and achieves strong predictive validity for offline evaluation, demonstrated by high correlation with real failure rates and low distributional divergence. Collectively, these results support adversarial, domain-specific simulation as a practical tool for rapid, reliable, and safe offline evaluation and potential RL-based optimization of mental health dialogue systems.

Abstract

Realistic user simulation is crucial for training and evaluating task-oriented dialogue (TOD) systems, yet creating simulators that accurately replicate human behavior remains challenging. A key property of effective simulators is their ability to expose failure modes of the systems they evaluate. We present an adversarial training framework that iteratively improves user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. Applied to mental health support chatbots, our approach demonstrates that fine-tuned simulators dramatically outperform zero-shot base models at surfacing system issues, and adversarial training further enhances diversity, distributional alignment, and predictive validity. The resulting simulator achieves a strong correlation between simulated and real failure occurrence rates across diverse chatbot configurations while maintaining low distributional divergence of failure modes. Discriminator accuracy decreases drastically after three adversarial iterations, suggesting improved realism. These results provide evidence that adversarial training is a promising approach for creating realistic user simulators in mental health support TOD domains, enabling rapid, reliable, and cost-effective system evaluation before deployment.

Figures (6)

• Figure 1: Overview of the adversarial training process. The simulator generates sessions that are evaluated by the discriminator to distinguish real from simulated conversations. Discriminator feedback guides preference dataset generation, which is then used to refine the simulator through DPO, creating an iterative competitive dynamic that improves realism.
• Figure 2: Issue rate comparison across simulated and real sessions. Error bars represent standard error. Zero-shot base models significantly underestimate issue rates, while fine-tuned models achieve realistic calibration.
• Figure 3: t-SNE visualization of session embeddings based on all messages. Production sessions are marked with stars. Base models (circles) form distinct clusters separate from real conversations, while fine-tuned simulators (squares) cluster tightly around their corresponding real sessions, demonstrating effective context grounding.
• Figure 4: Issue category discrepancy between UserSim-160K-it3 and real conversations. Left: categories over-represented in simulation (positive delta indicates simulation detects more). Right: categories under-represented in simulation (negative delta indicates real sessions have more). Deltas shown as percentage point differences.
• Figure 5: User action distributions comparing simulators to real data. Left: negative mental health indicators (Bad). Right: positive mental health indicators (Good). Error bars represent standard error. Zero-shot base models (GPT-4o, Kimi-K2-Instruct, Llama-3.3-70B-Instruct) show severe distributional misalignment, while fine-tuned simulators closely match real distributions.