Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structure Matters: Evaluating Multi-Agents Orchestration in Generative Therapeutic Chatbots

Sina Elahimanesh, Mohammadali Mohammadkhani, Sara Zahedi Movahed, Mohammadmahdi Abootorabi, Shayan Salehi, Abbas Edalat

TL;DR

It is demonstrated that for therapeutic AI, architectural orchestration is as critical as prompt engineering in fostering natural, engaging dialogue in fostering natural, engaging dialogue.

Abstract

While large language models (LLMs) excel at open-ended dialogue, effective psychotherapy requires structured progression and adherence to clinical protocols, making the design of psychotherapist chatbots challenging. We investigate how different LLM-based designs shape perceived therapeutic dialogue in a chatbot grounded in the Self-Attachment Technique (SAT), a novel self-administered psychotherapy rooted in attachment theory. We compare three architectural variants: (1) a multi-agent system utilizing finite state machine aligned with therapeutic stages and a shared long-term memory, (2) a single-agent using identical knowledge-base and the same prompts, and (3) an unguided LLM. In an eight-day randomized controlled trial (RCT) with N=66 Farsi-speaking participants, balanced across the three chatbots, the multi-agent system is perceived as significantly more natural and human-like than the other variants and achieves higher ratings across most other metrics. These findings demonstrate that for therapeutic AI, architectural orchestration is as critical as prompt engineering in fostering natural, engaging dialogue.

Structure Matters: Evaluating Multi-Agents Orchestration in Generative Therapeutic Chatbots

TL;DR

It is demonstrated that for therapeutic AI, architectural orchestration is as critical as prompt engineering in fostering natural, engaging dialogue in fostering natural, engaging dialogue.

Abstract

While large language models (LLMs) excel at open-ended dialogue, effective psychotherapy requires structured progression and adherence to clinical protocols, making the design of psychotherapist chatbots challenging. We investigate how different LLM-based designs shape perceived therapeutic dialogue in a chatbot grounded in the Self-Attachment Technique (SAT), a novel self-administered psychotherapy rooted in attachment theory. We compare three architectural variants: (1) a multi-agent system utilizing finite state machine aligned with therapeutic stages and a shared long-term memory, (2) a single-agent using identical knowledge-base and the same prompts, and (3) an unguided LLM. In an eight-day randomized controlled trial (RCT) with N=66 Farsi-speaking participants, balanced across the three chatbots, the multi-agent system is perceived as significantly more natural and human-like than the other variants and achieves higher ratings across most other metrics. These findings demonstrate that for therapeutic AI, architectural orchestration is as critical as prompt engineering in fostering natural, engaging dialogue.
Paper Structure (21 sections, 1 figure, 5 tables)

This paper contains 21 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experiment and Results
  5. Discussion
  6. Conclusion
  7. Finite State Machine Specification
  8. Transition Mechanisms
  9. Detailed State Descriptions
  10. Phase I: Initiation and Assessment
  11. Phase II: Exploration (Trigger Processing)
  12. Phase III: Therapeutic Intervention
  13. Phase IV: Conclusion
  14. Screenshot of the Web-based Interface
  15. Extended Quantitative Analysis
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Screenshot of the web-based user interface of the chatbot. After logging in, users are directed to the home screen where they can start interacting with the chatbot. (A) shows the list of user messages and corresponding chatbot responses. (B) is the input area for composing and sending messages to the chatbot. (C) contains the logout button on the left and the button to restart the conversation on the right.