Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Model

TL;DR

This work tackles the deployment barriers of large foundational models by distilling the MI reflection generation capability of GPT-4 into smaller GPT-2 models, enabling privacy-preserving and owned deployments for therapeutic chatbots. The authors build a high-quality dataset from smoking-cessation transcripts, generate simple and complex reflections with GPT-4 in a zero-shot setup, and fine-tune GPT-2 students to imitate these reflections. They also demonstrate a GPT-4-based evaluation pipeline for MI-adherence and reflection-type classification, benchmarked against human review, achieving substantial agreement. The approach shows strong performance gains for distilled models (76–93% MI-adherence) and provides a scalable, evaluation-driven path for end-to-end task-specific distillation in sensitive health contexts.

Abstract

Large Foundational Language Models are capable of performing many tasks at a high level but are difficult to deploy in many applications because of their size and proprietary ownership. Many will be motivated to distill specific capabilities of foundational models into smaller models that can be owned and controlled. In the development of a therapeutic chatbot, we wish to distill a capability known as reflective listening, in which a therapist produces reflections of client speech. These reflections either restate what a client has said, or connect what was said to a relevant observation, idea or guess that encourages and guides the client to continue contemplation. In this paper, we present a method for distilling the generation of reflections from a Foundational Language Model (GPT-4) into smaller models. We first show that GPT-4, using zero-shot prompting, can generate reflections at near 100% success rate, superior to all previous methods. Using reflections generated by GPT-4, we fine-tune different sizes of the GPT-2 family. The GPT-2-small model achieves 83% success on a hold-out test set and the GPT-2 XL achieves 90% success. We also show that GPT-4 can help in the labor-intensive task of evaluating the quality of the distilled models, using it as a zero-shot classifier. Using triple-human review as a guide, the classifier achieves a Cohen-Kappa of 0.66, a substantial inter-rater reliability figure.

Figures (3)

• Figure 1: Knowledge Distillation Overview
• Figure 2: Reflection Data Generation
• Figure 3: Reflection Evaluation Pipeline