Socratic Reasoning Improves Positive Text Rewriting

TL;DR

The paper tackles the challenge of cognitive reframing by introducing SocraticReframe, a training framework that makes a model generate Socratic rationales before reframing negative thoughts. By augmenting three open-source datasets with synthetic, CBT-grounded rationales generated via GPT-4 and training models to produce a rationale-then-reframe sequence, the authors demonstrate improvements across automatic and human evaluations, including expert validation. Key findings show that Socratic rationales enhance transfer strength and content preservation, with positive REV measures indicating informativeness of the rationales, and that human evaluators prefer Socratic-based reframes over baselines and even ChatGPT in many cases. The work highlights the synergy between LLM reasoning and established psychotherapy techniques, offering a path toward interpretable, clinically-aligned assistive tools for cognitive reframing and therapist training, while acknowledging limitations related to synthetic data and long-term outcomes.

Abstract

Reframing a negative into a positive thought is at the crux of several cognitive approaches to mental health and psychotherapy that could be made more accessible by large language model-based solutions. Such reframing is typically non-trivial and requires multiple rationalization steps to uncover the underlying issue of a negative thought and transform it to be more positive. However, this rationalization process is currently neglected by both datasets and models which reframe thoughts in one step. In this work, we address this gap by augmenting open-source datasets for positive text rewriting with synthetically-generated Socratic rationales using a novel framework called \textsc{SocraticReframe}. SocraticReframe uses a sequence of question-answer pairs to rationalize the thought rewriting process. We show that such Socratic rationales significantly improve positive text rewriting for different open-source LLMs according to both automatic and human evaluations guided by criteria from psychotherapy research. We validate our framework and the synthetic rationalizations with expert judgements from domain experts and psychology students in an IRB-approved annotation study. Our findings highlight the potential of utilizing the synergy between LLM reasoning and established psychotherapy techniques to build assistive solutions for reframing negative thoughts.

Figures (9)

• Figure 1: We use Socratic rationales consisting of question-answer pairs to improve positive text rewriting by verbalising the rewriting process. Here, we illustrate this: given a negative thought, we generate Socratic rationales grounded in Cognitive Behavioral Therapy to reframe the original thought into a positive thought.
• Figure 2: Detailed illustration of our framework SocraticReframe. First, we generate Socratic rationales using GPT-4 with a few-shot prompt. Then, we use the generated Socratic rationale to train models for cognitive reframing. During inference, the model generates the Socratic rationale before reframing the negative thought.
• Figure 3: A sample Socratic rationale for an instance from COGREF sharma-etal-2023-cognitive generated with GPT-4 in a few-shot setting. We use clinical vignettes from psychology literature as few-shot exemplars.
• Figure 4: The Socratic rationales generated for all three datasets have positive $\textsc{REV}$ scores, meaning that they indeed provide useful information for text rewriting. Mean of the example-wise $\textsc{REV}$ values are plotted for all three datasets.
• Figure 5: Sentiment scores for the generated Socratic question-answer pairs show that the synthetic rationales progressivley get better to lead to an improved reframe.