Domain-Specific Improvement on Psychotherapy Chatbot Using Assistant
Cheng Kang, Daniel Novak, Katerina Urbanova, Yuqing Cheng, Yong Hu
TL;DR
This paper tackles the challenge of domain-specific performance for psychotherapy tasks by LLMs, arguing that generic instruction data are insufficient for professional contexts. It introduces Domain-Specific Assistant Instructions and leverages a two-stage workflow—adaption fine-tuning and retrieval-augmented generation—guided by an assistant LLM (GPT-4) to infuse psychotherapy knowledge into pre-trained models. Using the Alexander Street psychotherapy transcript dataset, the authors train and evaluate models with inhibition adaptation fine-tuning and self-RAG, validated by both automatic metrics and human expert judgments, demonstrating substantial improvements over baselines. The work provides a practical, scalable pathway to align large language models with psychotherapy expertise, with implications for safer, more effective therapeutic chatbots and domain-specific AI assistants.
Abstract
Large language models (LLMs) have demonstrated impressive generalization capabilities on specific tasks with human-written instruction data. However, the limited quantity, diversity, and professional expertise of such instruction data raise concerns about the performance of LLMs in psychotherapy tasks when provided with domain-specific instructions. To address this, we firstly propose Domain-Specific Assistant Instructions based on AlexanderStreet therapy, and secondly, we use an adaption fine-tuning method and retrieval augmented generation method to improve pre-trained LLMs. Through quantitative evaluation of linguistic quality using automatic and human evaluation, we observe that pre-trained LLMs on Psychotherapy Assistant Instructions outperform state-of-the-art LLMs response baselines. Our Assistant-Instruction approach offers a half-annotation method to align pre-trained LLMs with instructions and provide pre-trained LLMs with more psychotherapy knowledge.