Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs

Anqi Li, Ruihan Wang, Zhaoming Chen, Yuqian Chen, Yu Lu, Yi Zhu, Yuan Xie, Zhenzhong Lan

Abstract

Effectively addressing client resistance is a sophisticated clinical skill in psychological counseling, yet practitioners often lack timely and scalable supervisory feedback to refine their approaches. Although current NLP research has examined overall counseling quality and general therapeutic skills, it fails to provide granular evaluations of high-stakes moments where clients exhibit resistance. In this work, we present a comprehensive pipeline for the multi-dimensional evaluation of human counselors' interventions specifically targeting client resistance in text-based therapy. We introduce a theory-driven framework that decomposes counselor responses into four distinct communication mechanisms. Leveraging this framework, we curate and share an expert-annotated dataset of real-world counseling excerpts, pairing counselor-client interactions with professional ratings and explanatory rationales. Using this data, we perform full-parameter instruction tuning on a Llama-3.1-8B-Instruct backbone to model fine-grained evaluative judgments of response quality and generate explanations underlying. Experimental results show that our approach can effectively distinguish the quality of different communication mechanisms (77-81% F1), substantially outperforming GPT-4o and Claude-3.5-Sonnet (45-59% F1). Moreover, the model produces high-quality explanations that closely align with expert references and receive near-ceiling ratings from human experts (2.8-2.9/3.0). A controlled experiment with 43 counselors further confirms that receiving these AI-generated feedback significantly improves counselors' ability to respond effectively to client resistance.

Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs

Abstract

Effectively addressing client resistance is a sophisticated clinical skill in psychological counseling, yet practitioners often lack timely and scalable supervisory feedback to refine their approaches. Although current NLP research has examined overall counseling quality and general therapeutic skills, it fails to provide granular evaluations of high-stakes moments where clients exhibit resistance. In this work, we present a comprehensive pipeline for the multi-dimensional evaluation of human counselors' interventions specifically targeting client resistance in text-based therapy. We introduce a theory-driven framework that decomposes counselor responses into four distinct communication mechanisms. Leveraging this framework, we curate and share an expert-annotated dataset of real-world counseling excerpts, pairing counselor-client interactions with professional ratings and explanatory rationales. Using this data, we perform full-parameter instruction tuning on a Llama-3.1-8B-Instruct backbone to model fine-grained evaluative judgments of response quality and generate explanations underlying. Experimental results show that our approach can effectively distinguish the quality of different communication mechanisms (77-81% F1), substantially outperforming GPT-4o and Claude-3.5-Sonnet (45-59% F1). Moreover, the model produces high-quality explanations that closely align with expert references and receive near-ceiling ratings from human experts (2.8-2.9/3.0). A controlled experiment with 43 counselors further confirms that receiving these AI-generated feedback significantly improves counselors' ability to respond effectively to client resistance.
Paper Structure (21 sections, 2 figures, 3 tables)

This paper contains 21 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation Framework
  4. Data Collection
  5. Data Source
  6. Annotation Task and Process
  7. Annotators Recruitment and Training
  8. Conversational Mechanisms Annotation
  9. Explanations Crafting
  10. Quality Control
  11. Privacy and Ethics
  12. Experiment and Results
  13. Problem Definition
  14. Experiment Setup
  15. Baselines
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Overview of our framework for evaluating counselor responses to client resistance. The framework comprises four core communication mechanisms: Respect for Autonomy, Stance Alignment, Emotional Resonance, and Conversational Orientation. For each mechanism, responses are further categorized into three levels of expression: no expression, weak expression, and strong expression. Our computational approach simultaneously identifies the strength of these dimensions along with their underlying explanations.
  • Figure 2: Interaction effects between experimental groups and phases across four dimensions. Solid green lines represent the control group, while dashed orange lines represent the experimental group. Points denote the mean values, and error bars indicate 95% confidence intervals. The results reveal a significant performance surge in the Experimental Group across all dimensions following the intervention, whereas the Control Group maintains a stable baseline.