Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ERD: A Framework for Improving LLM Reasoning for Cognitive Distortion Classification

Sehee Lim, Yejin Kim, Chi-Hyun Choi, Jy-yong Sohn, Byung-Hoon Kim

TL;DR

ERD is proposed, which improves LLM-based cognitive distortion classification performance with the aid of additional modules of extracting the parts related to cognitive distortion, and debating the reasoning steps by multiple agents.

Abstract

Improving the accessibility of psychotherapy with the aid of Large Language Models (LLMs) is garnering a significant attention in recent years. Recognizing cognitive distortions from the interviewee's utterances can be an essential part of psychotherapy, especially for cognitive behavioral therapy. In this paper, we propose ERD, which improves LLM-based cognitive distortion classification performance with the aid of additional modules of (1) extracting the parts related to cognitive distortion, and (2) debating the reasoning steps by multiple agents. Our experimental results on a public dataset show that ERD improves the multi-class F1 score as well as binary specificity score. Regarding the latter score, it turns out that our method is effective in debiasing the baseline method which has high false positive rate, especially when the summary of multi-agent debate is provided to LLMs.

ERD: A Framework for Improving LLM Reasoning for Cognitive Distortion Classification

TL;DR

ERD is proposed, which improves LLM-based cognitive distortion classification performance with the aid of additional modules of extracting the parts related to cognitive distortion, and debating the reasoning steps by multiple agents.

Abstract

Improving the accessibility of psychotherapy with the aid of Large Language Models (LLMs) is garnering a significant attention in recent years. Recognizing cognitive distortions from the interviewee's utterances can be an essential part of psychotherapy, especially for cognitive behavioral therapy. In this paper, we propose ERD, which improves LLM-based cognitive distortion classification performance with the aid of additional modules of (1) extracting the parts related to cognitive distortion, and (2) debating the reasoning steps by multiple agents. Our experimental results on a public dataset show that ERD improves the multi-class F1 score as well as binary specificity score. Regarding the latter score, it turns out that our method is effective in debiasing the baseline method which has high false positive rate, especially when the summary of multi-agent debate is provided to LLMs.
Paper Structure (9 sections, 2 figures, 4 tables)

This paper contains 9 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. ERD
  3. Extraction
  4. Reasoning
  5. Debate
  6. Experiments
  7. Settings
  8. Experimental results
  9. Conclusion

Figures (2)

  • Figure 1: The pipeline of Extraction-Reasoning-Debate (ERD), which detects and classify the cognitive distortion from the input user speech. It begins with the identification and extraction of potential cognitive distortions from the user speech. These extracted elements are then utilized to construct an intermediate reasoning step. Subsequently, a debate is conducted, wherein multiple LLM agents deliberate to assess the presence and type of cognitive distortion. Finally, a judge integrates the entire debate process to get the final answer on the distortion classification problem.
  • Figure 2: Confusion matrices of ERD when tested on 2530 samples: (Left) only Reasoning is used, (Right) Extraction, Reasoning and Debate steps are used. Including Extraction and Debate modules increases the number of true negatives from 61 to 322, thus correctly identifying the samples with 'no distortion'.