Estimating Quality in Therapeutic Conversations: A Multi-Dimensional Natural Language Processing Framework

TL;DR

Therapy engagement is a critical determinant of outcomes but lacks objective measurement in many settings. The authors introduce a multi-dimensional NLP framework that analyzes therapy transcripts across four feature domains—conversational dynamics, semantic similarity, sentiment, and question detection—and trains RF, CatBoost, and SVM classifiers with SMOTE-Tomek augmentation. They achieve strong performance, with up to 88.9% accuracy and 94.6% AUC on augmented data, and reveal that semantic similarity and topic alignment are the most predictive features via SHAP explanations. The work demonstrates a scalable, data-driven approach for real-time engagement assessment that can augment clinical practice and motivate future multimodal extensions to further improve psychotherapy outcomes.

Abstract

Engagement between client and therapist is a critical determinant of therapeutic success. We propose a multi-dimensional natural language processing (NLP) framework that objectively classifies engagement quality in counseling sessions based on textual transcripts. Using 253 motivational interviewing transcripts (150 high-quality, 103 low-quality), we extracted 42 features across four domains: conversational dynamics, semantic similarity as topic alignment, sentiment classification, and question detection. Classifiers, including Random Forest (RF), Cat-Boost, and Support Vector Machines (SVM), were hyperparameter tuned and trained using a stratified 5-fold cross-validation and evaluated on a holdout test set. On balanced (non-augmented) data, RF achieved the highest classification accuracy (76.7%), and SVM achieved the highest AUC (85.4%). After SMOTE-Tomek augmentation, performance improved significantly: RF achieved up to 88.9% accuracy, 90.0% F1-score, and 94.6% AUC, while SVM reached 81.1% accuracy, 83.1% F1-score, and 93.6% AUC. The augmented data results reflect the potential of the framework in future larger-scale applications. Feature contribution revealed conversational dynamics and semantic similarity between clients and therapists were among the top contributors, led by words uttered by the client (mean and standard deviation). The framework was robust across the original and augmented datasets and demonstrated consistent improvements in F1 scores and recall. While currently text-based, the framework supports future multimodal extensions (e.g., vocal tone, facial affect) for more holistic assessments. This work introduces a scalable, data-driven method for evaluating engagement quality of the therapy session, offering clinicians real-time feedback to enhance the quality of both virtual and in-person therapeutic interactions.

Figures (4)

• Figure 1: Overview of the conversation quality assessment framework. Diarization timing and terms were extracted from the available videos. The system processes transcripts through four feature domains (conversational dynamics, semantic similarity, sentiment, and question detection), followed by data processing steps, before classification using Random Forest, CatBoost and SVM. Finishing with performance measure.
• Figure 2: Feature correlation heatmap with highest value of 1 and lowest value of -0.71.
• Figure 3: Performance measure illustrating the holdout test set on the cross-validation folds for Random Forest (RF), CatBoost (CB) and Support Vector Machine (SVM). The holdout test set has only 18 samples, with 9 from each class. Instead of providing the best result, the test set was used in each cross-validation fold after hyperparameter tuning.
• Figure 4: Kernel density estimation of the balanced features using SMOTE+Tomek.