Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ESC-Judge: A Framework for Comparing Emotional Support Conversational Agents

Navid Madani, Rohini Srihari

TL;DR

The paper tackles the lack of scalable, theory-grounded evaluation for emotional-support LLMs. It introduces ESC-Judge, a three-stage framework anchored in Clara Hill's Exploration–Insight–Action model that automates role construction, parallel dialogue simulation, and rubric-based judging to compare ES agents head-to-head. A specialized judge LLM achieves substantial alignment with human annotators (about 86% for Exploration, 83% for Insight, and 85% for Action) while reducing annotation costs, demonstrating human-level reliability at scale. The framework is open-source and designed to enable scalable optimization of ES agents while maintaining theoretical grounding and transparency for safer deployment in high-stakes contexts.

Abstract

Large language models (LLMs) increasingly power mental-health chatbots, yet the field still lacks a scalable, theory-grounded way to decide which model is most effective to deploy. We present ESC-Judge, the first end-to-end evaluation framework that (i) grounds head-to-head comparisons of emotional-support LLMs in Clara Hill's established Exploration-Insight-Action counseling model, providing a structured and interpretable view of performance, and (ii) fully automates the evaluation pipeline at scale. ESC-Judge operates in three stages: first, it synthesizes realistic help-seeker roles by sampling empirically salient attributes such as stressors, personality, and life history; second, it has two candidate support agents conduct separate sessions with the same role, isolating model-specific strategies; and third, it asks a specialized judge LLM to express pairwise preferences across rubric-anchored skills that span the Exploration, Insight, and Action spectrum. In our study, ESC-Judge matched PhD-level annotators on 85 percent of Exploration, 83 percent of Insight, and 86 percent of Action decisions, demonstrating human-level reliability at a fraction of the cost. All code, prompts, synthetic roles, transcripts, and judgment scripts are released to promote transparent progress in emotionally supportive AI.

ESC-Judge: A Framework for Comparing Emotional Support Conversational Agents

TL;DR

The paper tackles the lack of scalable, theory-grounded evaluation for emotional-support LLMs. It introduces ESC-Judge, a three-stage framework anchored in Clara Hill's Exploration–Insight–Action model that automates role construction, parallel dialogue simulation, and rubric-based judging to compare ES agents head-to-head. A specialized judge LLM achieves substantial alignment with human annotators (about 86% for Exploration, 83% for Insight, and 85% for Action) while reducing annotation costs, demonstrating human-level reliability at scale. The framework is open-source and designed to enable scalable optimization of ES agents while maintaining theoretical grounding and transparency for safer deployment in high-stakes contexts.

Abstract

Large language models (LLMs) increasingly power mental-health chatbots, yet the field still lacks a scalable, theory-grounded way to decide which model is most effective to deploy. We present ESC-Judge, the first end-to-end evaluation framework that (i) grounds head-to-head comparisons of emotional-support LLMs in Clara Hill's established Exploration-Insight-Action counseling model, providing a structured and interpretable view of performance, and (ii) fully automates the evaluation pipeline at scale. ESC-Judge operates in three stages: first, it synthesizes realistic help-seeker roles by sampling empirically salient attributes such as stressors, personality, and life history; second, it has two candidate support agents conduct separate sessions with the same role, isolating model-specific strategies; and third, it asks a specialized judge LLM to express pairwise preferences across rubric-anchored skills that span the Exploration, Insight, and Action spectrum. In our study, ESC-Judge matched PhD-level annotators on 85 percent of Exploration, 83 percent of Insight, and 86 percent of Action decisions, demonstrating human-level reliability at a fraction of the cost. All code, prompts, synthetic roles, transcripts, and judgment scripts are released to promote transparent progress in emotionally supportive AI.
Paper Structure (38 sections, 4 equations, 12 figures, 7 tables)

This paper contains 38 sections, 4 equations, 12 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Road-map.
  3. Related Work
  4. Human Evaluation of Emotional-Support Dialogues
  5. Automated and LLM-Based Evaluation Protocols
  6. ESC-Judge
  7. Framework Overview
  8. Patient Role Construction
  9. Ongoing Challenge and Stressor.
  10. Demographic Information.
  11. Key Life Events.
  12. Behavioral Traits
  13. Simulate Emotional Support Conversation
  14. Dialogue engine
  15. Turn budget and early stopping
  16. ...and 23 more sections

Figures (12)

  • Figure 1: Overall pipeline of our proposed ES-Judge framework. Stage 1: constructs a diverse set of roles with various life backgrounds, demographics and behavioral attributes. Stage 2: conditioning on a fixed help seeker role, we have two emotional support (ES) models under test to participate in an emotional support conversation and we store the conversation transcripts. Stage 3: given carefully curated evaluation dimensions based on Hill's framework, we compare the capabilities of the two models under test on performing Exploration, Insight and Action.
  • Figure 2: Role construction agents: Orange agents are random samplers based on pre-defined categories. Blue agents use generative prompts to explore the desired domain. The green agent only validates and compiles the final role without adding new information. Arrows represent the flow of data between agents.
  • Figure 3: Left and right columns represent the first 7 turns of conversation between one help seeker role and two emotional support agents. One left ES agent is llama3.2-3b-instruct with Hill's guideline prompt and on the right we have GPT-4o without any guidelines as ES agent. ESC-Judge marks the left agent as the winner on exploration category.
  • Figure 4: Comparison of the win-rate of different ES agents according to ESC-Judge framwork on three stages of exploration, insight and actoin.
  • Figure 5: A full sample role from the role construction pipeline
  • ...and 7 more figures