Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations

TL;DR

An LLM-based Multi-User Simulator (MUS) is proposed to ease MUCA's optimization, enabling faster simulation of conversations between the chatbot and simulated users, and speeding up MUCA's early development.

Abstract

Recent advancements in large language models (LLMs) have provided a new avenue for chatbot development. Most existing research, however, has primarily centered on single-user chatbots that determine "What" to answer. This paper highlights the complexity of multi-user chatbots, introducing the 3W design dimensions: "What" to say, "When" to respond, and "Who" to answer. Additionally, we proposed Multi-User Chat Assistant (MUCA), an LLM-based framework tailored for group discussions. MUCA consists of three main modules: Sub-topic Generator, Dialog Analyzer, and Conversational Strategies Arbitrator. These modules jointly determine suitable response contents, timings, and appropriate addressees. This paper further proposes an LLM-based Multi-User Simulator (MUS) to ease MUCA's optimization, enabling faster simulation of conversations between the chatbot and simulated users, and speeding up MUCA's early development. In goal-oriented conversations with a small to medium number of participants, MUCA demonstrates effectiveness in tasks like chiming in at appropriate timings, generating relevant content, and improving user engagement, as shown by case studies and user studies.

Figures (6)

• Figure 1: A diagram mapping out a group chat sample to its associated five design challenges and further formulated to the proposed 3W design dimensions.
• Figure 2: Framework architecture, which is composed of the proposed MUCA (Sec. \ref{['subsec:framework_architecture']}) and MUS (Sec. \ref{['subsec:user_simu']}). The MUCA is periodically iterated via the interaction with the proposed MUS in the development mode, while real users are interacting with MUCA in the evaluation mode. The temporary results in the gray dash boxes serve as examples.
• Figure 3: Qualitative comparison between Baseline-small and MUCA-small: A), B) hallucination issues, C) summarization feature, and D) conflict resolution capability. The conversation consists of 1 chatbot (_bot_Spirit for Baseline-small or _bot_Perseverance for MUCA-small) and 4 participants, namely, Amy, Bob, Cindy, and Dennis.
• Figure 4: A comparison between Baseline-small and MUCA-small. Each set of results presents the performance of Baseline-small and MUCA-small in two separate rows. In (a)-(c), each bar chart illustrates the counts of options selected by users if they ever encountered these scenarios during the chat. The accompanying statistics on the right-hand side summarize the counts in each row. In (d), users rated each chatbot on efficiency, conciseness, and usefulness, using options from "Very Good" to "Very Poor". Corresponding scores are displayed on the right.
• Figure 5: Data flow for Dialog Analyzer, which includes participant feature extractor and three LLM-based modules -- sub-topic status update, utterance feature extractor, and accumulative summary update. The placeholders (printed in purple) in the prompts are filled by sub-topics from the Sub-topic Generator, conversation signals such as attendee names, and utterances in $U_{N_{sw}, i}$. The generated outputs (sub-topic status, accumulative summary, and sub-topic being discussed, all printed in yellow) will be fed back to the sub-topics status update and accumulative summary update as inputs for the execution in the next round.