Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis

Elham Aghakhani, Lu Wang, Karla T. Washington, George Demiris, Jina Huh-Yoo, Rezvaneh Rezapour

TL;DR

The paper investigates automating the annotation of problem-solving therapy (PST) dialogues using large language models (LLMs) and transformer models. It introduces a PST annotation framework combining core PST strategies with novel Facilitative Strategies and evaluates multiple models on 240 real PST transcripts, revealing GPT-4o as the top annotator (F1 ≈ 0.76) without context, while transformer models can achieve high performance with domain-specific fine-tuning. The study also analyzes Therapeutic Dynamics (autonomy, self-disclosure, metaphor) and linguistic patterns via LIWC to understand how strategies are applied and how sessions progress. These findings support scalable, privacy-conscious PST analysis and point toward human-in-the-loop AI assistance to enhance therapy research and practice.

Abstract

Problem-solving therapy (PST) is a structured psychological approach that helps individuals manage stress and resolve personal issues by guiding them through problem identification, solution brainstorming, decision-making, and outcome evaluation. As mental health care increasingly adopts technologies like chatbots and large language models (LLMs), it is important to thoroughly understand how each session of PST is conducted before attempting to automate it. We developed a comprehensive framework for PST annotation using established PST Core Strategies and a set of novel Facilitative Strategies to analyze a corpus of real-world therapy transcripts to determine which strategies are most prevalent. Using various LLMs and transformer-based models, we found that GPT-4o outperformed all models, achieving the highest accuracy (0.76) in identifying all strategies. To gain deeper insights, we examined how strategies are applied by analyzing Therapeutic Dynamics (autonomy, self-disclosure, and metaphor), and linguistic patterns within our labeled data. Our research highlights LLMs' potential to automate therapy dialogue analysis, offering a scalable tool for mental health interventions. Our framework enhances PST by improving accessibility, effectiveness, and personalized support for therapists.

From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis

TL;DR

The paper investigates automating the annotation of problem-solving therapy (PST) dialogues using large language models (LLMs) and transformer models. It introduces a PST annotation framework combining core PST strategies with novel Facilitative Strategies and evaluates multiple models on 240 real PST transcripts, revealing GPT-4o as the top annotator (F1 ≈ 0.76) without context, while transformer models can achieve high performance with domain-specific fine-tuning. The study also analyzes Therapeutic Dynamics (autonomy, self-disclosure, metaphor) and linguistic patterns via LIWC to understand how strategies are applied and how sessions progress. These findings support scalable, privacy-conscious PST analysis and point toward human-in-the-loop AI assistance to enhance therapy research and practice.

Abstract

Problem-solving therapy (PST) is a structured psychological approach that helps individuals manage stress and resolve personal issues by guiding them through problem identification, solution brainstorming, decision-making, and outcome evaluation. As mental health care increasingly adopts technologies like chatbots and large language models (LLMs), it is important to thoroughly understand how each session of PST is conducted before attempting to automate it. We developed a comprehensive framework for PST annotation using established PST Core Strategies and a set of novel Facilitative Strategies to analyze a corpus of real-world therapy transcripts to determine which strategies are most prevalent. Using various LLMs and transformer-based models, we found that GPT-4o outperformed all models, achieving the highest accuracy (0.76) in identifying all strategies. To gain deeper insights, we examined how strategies are applied by analyzing Therapeutic Dynamics (autonomy, self-disclosure, and metaphor), and linguistic patterns within our labeled data. Our research highlights LLMs' potential to automate therapy dialogue analysis, offering a scalable tool for mental health interventions. Our framework enhances PST by improving accessibility, effectiveness, and personalized support for therapists.
Paper Structure (40 sections, 1 equation, 10 figures, 8 tables)

This paper contains 40 sections, 1 equation, 10 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Mental Health Care
  4. LLMs as (Co-)Annotators
  5. Method
  6. Data
  7. Classification of PST Strategies
  8. Therapeutic Dynamics
  9. Linguistic Patterns
  10. Result
  11. Performance of LLMs in Annotation
  12. Transformer-Based Models
  13. Therapeutic Dynamics
  14. Linguistic Patterns
  15. Strategic Progression in Therapy Sessions.
  16. ...and 25 more sections

Figures (10)

  • Figure 1: A real-life example of therapist and client exchanges in a problem-solving therapy session.
  • Figure 2: Distribution of Autonomy in PS Core
  • Figure 3: Distribution of Autonomy in Facilitators
  • Figure 4: Proportion of Metaphors in PS Core
  • Figure 5: Proportion of Metaphors in Facilitators
  • ...and 5 more figures