Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Therapeutic Relationship between Counselors and Clients in Online Text-based Counseling using LLMs

Anqi Li, Yu Lu, Nirui Song, Shuai Zhang, Lizhi Ma, Zhenzhong Lan

TL;DR

This paper adapts a theoretically grounded framework specifically to the context of online text-based counseling and develops comprehensive guidelines for characterizing the alliance and demonstrates effectiveness in identifying the therapeutic alliance.

Abstract

Robust therapeutic relationships between counselors and clients are fundamental to counseling effectiveness. The assessment of therapeutic alliance is well-established in traditional face-to-face therapy but may not directly translate to text-based settings. With millions of individuals seeking support through online text-based counseling, understanding the relationship in such contexts is crucial. In this paper, we present an automatic approach using large language models (LLMs) to understand the development of therapeutic alliance in text-based counseling. We adapt a theoretically grounded framework specifically to the context of online text-based counseling and develop comprehensive guidelines for characterizing the alliance. We collect a comprehensive counseling dataset and conduct multiple expert evaluations on a subset based on this framework. Our LLM-based approach, combined with guidelines and simultaneous extraction of supportive evidence underlying its predictions, demonstrates effectiveness in identifying the therapeutic alliance. Through further LLM-based evaluations on additional conversations, our findings underscore the challenges counselors face in cultivating strong online relationships with clients. Furthermore, we demonstrate the potential of LLM-based feedback mechanisms to enhance counselors' ability to build relationships, supported by a small-scale proof-of-concept.

Understanding the Therapeutic Relationship between Counselors and Clients in Online Text-based Counseling using LLMs

TL;DR

This paper adapts a theoretically grounded framework specifically to the context of online text-based counseling and develops comprehensive guidelines for characterizing the alliance and demonstrates effectiveness in identifying the therapeutic alliance.

Abstract

Robust therapeutic relationships between counselors and clients are fundamental to counseling effectiveness. The assessment of therapeutic alliance is well-established in traditional face-to-face therapy but may not directly translate to text-based settings. With millions of individuals seeking support through online text-based counseling, understanding the relationship in such contexts is crucial. In this paper, we present an automatic approach using large language models (LLMs) to understand the development of therapeutic alliance in text-based counseling. We adapt a theoretically grounded framework specifically to the context of online text-based counseling and develop comprehensive guidelines for characterizing the alliance. We collect a comprehensive counseling dataset and conduct multiple expert evaluations on a subset based on this framework. Our LLM-based approach, combined with guidelines and simultaneous extraction of supportive evidence underlying its predictions, demonstrates effectiveness in identifying the therapeutic alliance. Through further LLM-based evaluations on additional conversations, our findings underscore the challenges counselors face in cultivating strong online relationships with clients. Furthermore, we demonstrate the potential of LLM-based feedback mechanisms to enhance counselors' ability to build relationships, supported by a small-scale proof-of-concept.
Paper Structure (53 sections, 6 figures, 9 tables)

This paper contains 53 sections, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Evaluation of Counseling Using NLP.
  4. LLMs for Mental Health Analysis.
  5. Framework and Guidelines for Measuring Therapeutic Alliance
  6. Framework
  7. Goal.
  8. Approach.
  9. Affective Bond.
  10. Guidelines
  11. Data Collection
  12. Data Source
  13. Annotation Process
  14. Data Characteristics
  15. LLM Evaluation
  16. ...and 38 more sections

Figures (6)

  • Figure 1: Our therapeutic alliance framework comprises three integral components: consensus on goal-setting and approaches, and the cultivation of affective bonds. Each component is measured by four questions, each scored with customized guidelines, distinguishing between substantial evidence against, some evidence against, no evidence against, some evidence for, and substantial evidence for these aspects.
  • Figure 2: The violin plot of the distribution of scores annotated for each question, with a boxplot inside. The white pentagons within the violins represent the mean values.
  • Figure 3: The average alliance scores for all counselors and counselors with varied experience levels.
  • Figure 4: The average alliance across counseling stages.
  • Figure 5: Example prompts for evaluating a giving conversation across different experimental setups (i.e. with different prompt types and with/without CoT) addressing question There is agreement about the steps taken to help improve the client's situation. General guidelines remain consistent across different questions, whereas detailed guidelines are intricately tailored to each specific question.
  • ...and 1 more figures