ChoiceMates: Supporting Unfamiliar Online Decision-Making with Multi-Agent Conversational Interactions

TL;DR

ChoiceMates introduces a user-operated, multi-agent conversational system designed to support unfamiliar online decisions by enabling users to interact with a dynamic set of agent personas. Through a formative study and a within-subject user study, the approach demonstrates that surface-level breadth of information and structured preference building can be achieved with lower cognitive burden and higher confidence than traditional web search, while outperforming a commercial multi-agent baseline in terms of final decision quality and user satisfaction. The system combines a conversation space, a history log, and a preference space, aided by a lightweight retrieval-augmented generation pipeline and multi-agent prompting that preserves user agency and enables inter-agent debates. The work highlights design considerations for controllable, collaborative AI-based decision support and discusses limitations related to hallucinations and the need for robust validation in real-world deployments.

Abstract

From deciding on a PhD program to buying a new camera, unfamiliar decisions--decisions without domain knowledge--are frequent and significant. The complexity and uncertainty of such decisions demand unique approaches to information seeking, understanding, and decision-making. Our formative study highlights that users want to start by discovering broad and relevant domain information evenly and simultaneously, quickly address emerging inquiries, and gain personalized standards to assess information found. We present ChoiceMates, an interactive multi-agent system designed to address these needs by enabling users to engage with a dynamic set of LLM agents each presenting a unique experience in the domain. Unlike existing multi-agent systems that automate tasks with agents, the user orchestrates agents to assist their decision-making process. Our user evaluation (n=12) shows that ChoiceMates enables a more confident, satisfactory decision-making with better situation understanding than web search, and higher decision quality and confidence than a commercial multi-agent framework. This work provides insights into designing a more controllable and collaborative multi-agent system.

Figures (8)

• Figure 1: The ChoiceMates interface: (1) Agents populate the conversation space to converse with the user, identifying key criteria and options in their utterances (1). The Conversation History automatically lists those criteria and options mentioned by agents (2). Preference Space contains the options and criteria selected by the user from the Conversation History. (a) indicates the unfamiliar information stacked in the Conversation History. (b) shows the saving process of the user preference. Lastly, (c) illustrates the flow of applying user preference into the Conversation Space with the agents in ChoiceMates.
• Figure 2: (1) An agent utterance is represented as a chat bubble above the agent icon. (2) Hovering over the icon reveals the agent's profile containing their valued criteria and their chosen option.
• Figure 3: Three types of conversation interactions possible in the conversation space. (1) The user can chat with any agents currently in the space, (2) select or tag one or more agents to chat, and (3) call new agents into the conversation space with their input. Like the conversation in (2), the agents will build on top of each other's responses when applicable.
• Figure 4: Consequences of preference toggle. With the same user message "Any other agents?", when the toggle button is off (1), information in the preference space is hidden from the agents, thus making the responses independent from the user's preferences. When the toggle button is on (2) the responses are tailored to the preference of the user.
• Figure 5: Overview of the technical pipeline. When the user sends their message to ChoiceMates, the conversation context is sent to the LLM (1) and is instructed to infer the user's intent (2) and respond through the most relevant agents (3).