Emotional Support Evaluation Framework via Controllable and Diverse Seeker Simulator

TL;DR

This work tackles the unreliable evaluation of emotional support chatbots by introducing a controllable seeker simulator that captures diverse real-world seeker behaviors. Built on Reddit data and powered by a Mixture-of-Experts routing mechanism with LoRA-based adapters, the framework supports fine-grained control over seeker profiles across nine psychological and linguistic features. The authors demonstrate improved profile adherence, fidelity, and diversity, and reveal that supporter models exhibit varying degradation when evaluated against diverse seeker populations. The framework enables population-aware, stress-tested evaluation with actionable insights for improving emotional support systems, while noting limitations such as single-dialogue assessment and suggesting directions for future work on dynamic profiles and outcome-oriented metrics.

Abstract

As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged: use help-seeker simulators to interact with supporter chatbots. However, current simulators suffer from two critical limitations: (1) they fail to capture the behavioral diversity of real-world seekers, often portraying them as overly cooperative, and (2) they lack the controllability required to simulate specific seeker profiles. To address these challenges, we present a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior. Using authentic Reddit conversations, we train our model via a Mixture-of-Experts (MoE) architecture, which effectively differentiates diverse seeker behaviors into specialized parameter subspaces, thereby enhancing fine-grained controllability. Our simulator achieves superior profile adherence and behavioral diversity compared to existing approaches. Furthermore, evaluating 7 prominent supporter models with our system uncovers previously obscured performance degradations. These findings underscore the utility of our framework in providing a more faithful and stress-tested evaluation for emotional support chatbots.

Figures (10)

• Figure 1: Overview of our evaluation framework for emotional support models.
• Figure 2: Overview of the MoLA architecture. A frozen SFT backbone is augmented with multiple low-rank expert adapters at each linear layer. A shared routing network produces a dialogue-level routing vector $\boldsymbol{\alpha}$, which controls expert activation consistently across all layers.
• Figure 3: UMAP projection of dialogue-level seeker embeddings across simulators.
• Figure 4: Common system instruction for feature tagging.
• Figure 5: PCA projection of routing distributions ($\boldsymbol{\alpha}$), colored by dominant expert. The separation indicates that expert routing occupies distinct regions in routing space.