Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CounselReflect: A Toolkit for Auditing Mental-Health Dialogues

Yahan Li, Chaohao Du, Zeyang Li, Christopher Chun Kuizon, Shupeng Cheng, Angel Hsing-Chi Hwang, Adam C. Frank, Ruishan Liu

Abstract

Mental-health support is increasingly mediated by conversational systems (e.g., LLM-based tools), but users often lack structured ways to audit the quality and potential risks of the support they receive. We introduce CounselReflect, an end-to-end toolkit for auditing mental-health support dialogues. Rather than producing a single opaque quality score, CounselReflect provides structured, multi-dimensional reports with session-level summaries, turn-level scores, and evidence-linked excerpts to support transparent inspection. The system integrates two families of evaluation signals: (i) 12 model-based metrics produced by task-specific predictors, and (ii) rubric-based metrics that extend coverage via a literature-derived library (69 metrics) and user-defined custom metrics, operationalized with configurable LLM judges. CounselReflect is available as a web application, browser extension, and command-line interface (CLI), enabling use in real-time settings as well as at scale. Human evaluation includes a user study with 20 participants and an expert review with 6 mental-health professionals, suggesting that CounselReflect supports understandable, usable, and trustworthy auditing. A demo video and full source code are also provided.

CounselReflect: A Toolkit for Auditing Mental-Health Dialogues

Abstract

Mental-health support is increasingly mediated by conversational systems (e.g., LLM-based tools), but users often lack structured ways to audit the quality and potential risks of the support they receive. We introduce CounselReflect, an end-to-end toolkit for auditing mental-health support dialogues. Rather than producing a single opaque quality score, CounselReflect provides structured, multi-dimensional reports with session-level summaries, turn-level scores, and evidence-linked excerpts to support transparent inspection. The system integrates two families of evaluation signals: (i) 12 model-based metrics produced by task-specific predictors, and (ii) rubric-based metrics that extend coverage via a literature-derived library (69 metrics) and user-defined custom metrics, operationalized with configurable LLM judges. CounselReflect is available as a web application, browser extension, and command-line interface (CLI), enabling use in real-time settings as well as at scale. Human evaluation includes a user study with 20 participants and an expert review with 6 mental-health professionals, suggesting that CounselReflect supports understandable, usable, and trustworthy auditing. A demo video and full source code are also provided.

Paper Structure

This paper contains 45 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Overview of CounselReflect
  3. Deployment Modes.
  4. Core Components
  5. Model-Based Metrics
  6. Empathy
  7. Emotion Identification
  8. Emotional Trigger
  9. Reflection
  10. Supporter Strategy
  11. Talk Type
  12. Toxicity
  13. Factuality
  14. Predictor Performance
  15. Literature-Derived Metrics and Custom Metrics
  16. ...and 30 more sections

Figures (3)

  • Figure 1: CounselReflect workflow (shown in web interface). (1) Configure the model/API and upload a conversation transcript. (2) Select evaluation metrics, with support for metric customization and rubric refinement. (3) Explore results through interactive session-level summaries, metric visualizations, and turn-level analysis.
  • Figure 2: Left: Chrome extension overlay auditing a ChatGPT conversation with highlighted evidence and turn-level scores. Right: CLI for selecting metrics, running evaluation, and exporting structured results.
  • Figure 3: Post-study survey results across two participant groups. Dashed lines indicate reference thresholds. Bars show group means and error bars denote standard deviations.