Can LLM-Simulated Practice and Feedback Upskill Human Counselors? A Randomized Study with 90+ Novice Counselors

TL;DR

This study evaluates CARE, an LLM-based training system for novice counselors that combines AI-simulated practice with AI-generated feedback. In a randomized trial (N=94), practicing with feedback (P+F) improved empathetic listening, reflections, and client-centered questioning, whereas practice alone (P) did not and even reduced empathy. Self-efficacy calibration was generally misaligned with performance, highlighting limits of self-assessment as a performance proxy. The findings suggest that scalable, AI-driven counselor training should integrate both realistic practice and structured, skill-focused feedback to develop essential client-centered skills and support broader access to high-quality training. The work has practical implications for scalable mental health training, suggesting a hybrid, AI-enabled approach can complement traditional supervision and role-play methods.

Abstract

Training more counselors, from clinical students to peer supporters, can help meet the demand for accessible mental health support; however, current training approaches remain resource-intensive and difficult to scale effectively. Large Language Models (LLMs) offer promising solutions for growing counseling skills training through simulated practice and automated feedback. Despite successes in aligning LLMs with expert-counselor annotations, we do not know whether LLM-based counseling training tools -- such as AI patients that simulate real-world challenges and generative AI feedback with suggested alternatives and rationales -- actually lead to improvements in novice counselor skill development. We develop CARE, an LLM-simulated practice and feedback system, and randomize 94 novice counselors to practice using an AI patient, either alone or with AI feedback, measuring changes in their behavioral performance, self-assessments, and qualitative learning takeaways. Our results show the practice-and-feedback group improved in their use of reflections and questions (d=0.32-0.39, p$<$0.05). In contrast, the group that practiced with an AI patient alone did not show improvements, and in the case of empathy, actually had worse uses across time (d=$-$0.52, p=0.001) and when compared against the practice-and-feedback group (d=0.72, p=0.001). Participants' qualitative self-reflections revealed key differences: the practice-and-feedback group adopted a client-centered approach involving listening to and validating feelings, while the practice-alone group remained solution-oriented but delayed offering suggestions until gathering more information. Overall, these results suggest that LLM-based training systems can promote effective skill development, but that combining both simulated practice and structured feedback is critical.

Figures (6)

• Figure 1: Our between-subjects study randomizes participants to either practice with AI Patients alone (P) or practice with AI patients and receive AI feedback (P+F). We holistically evaluate counselor skill development from three perspectives: automatic assessments of behaviors of skills used via LLM classifiers; self-efficacy and its calibration with actual performance; and qualitative self-reflections after the training intervention chat and post-intervention chat.
• Figure 2: Changes in counseling behaviors following AI patient simulations alone (P) versus AI patient simulations with AI feedback (P+F). The plot displays bootstrapped means for pre-intervention and post-intervention interactions. The table presents statistical comparisons with corresponding effect sizes, with bolded values indicating significance after Benjamani-Hochberg correction benjamini1995controlling for the 21 planned comparisons (12 for behavioral changes and 9 for self-efficacy changes). Notably, P group experiences a significant drop in strong uses of Empathy (-9.6% change, $d=-0.52$), whereas P+F group's use of Empathy is maintained and trends towards improvement. The P+F group also experiences noticeable improvements in Reflections (+3.7% change, $d=0.32$) and Questions (6.59% change, $d=0.36$)
• Figure 3: Counselor Self Efficacy (perceived ability to use skills) for participants grouped by behaviors of skills used (actual performance). Notes: Gaps depict miscalibration between actual and self-assessed percentile of performance for quartile groups with bootstrapped 95% CIs. We only visualize for data collected in the post-intervention.
• Figure 4: Changes in raw-scores of self-efficacy following AI patient simulations alone (P) versus AI patient simulations with AI feedback (PF). The plot displays bootstrapped means for pre-intervention and post-intervention. The table presents statistical comparisons with corresponding effect sizes, with bolded values indicating significance after Benjamani-Hochberg correction benjamini1995controlling for the 21 planned comparisons (12 for behavioral changes and 9 for self-efficacy changes).
• Figure 5: CARE's practice and feedback model visualized in a diagram and web screenshot. In CARE, counselors practice with an LLM-simulated patient and receive feedback on each of their responses. The feedback model labels whether a response has strengths or constructive feedback areas. Responses with constructive feedback explain what the goal should be at this point in the conversation; what a helper could improve to better align with this goal; and how they could respond differently via an alternative response.