Users' Mental Models of Generative AI Chatbot Ecosystems

TL;DR

GenAI chatbot ecosystems introduce complex data flows across a network of entities (chatbots, parent companies, and plugins), raising privacy questions for users. The authors conduct 21 semi-structured interviews comparing first-party (Gemini) and third-party (ChatGPT) ecosystems, uncovering four mental models (Key Player, Medium, Representation, Agent) and revealing higher trust in third-party ecosystems due to more transparent cues. They find that first-party ecosystems tend to provoke more privacy concerns due to integration opacity, while third-party ecosystems benefit from perceptible plugin cues but still require clear data-flow transparency. The study offers design and policy implications, including explicit privacy notices, user controls, and regulatory disclosure of involved entities to improve user understanding and privacy protection in GenAI ecosystems.

Abstract

The capability of GenAI-based chatbots, such as ChatGPT and Gemini, has expanded quickly in recent years, turning them into GenAI Chatbot Ecosystems. Yet, users' understanding of how such ecosystems work remains unknown. In this paper, we investigate users' mental models of how GenAI Chatbot Ecosystems work. This is an important question because users' mental models guide their behaviors, including making decisions that impact their privacy. Through 21 semi-structured interviews, we uncovered users' four mental models towards first-party (e.g., Google Gemini) and third-party (e.g., ChatGPT) GenAI Chatbot Ecosystems. These mental models centered around the role of the chatbot in the entire ecosystem. We further found that participants held a more consistent and simpler mental model towards third-party ecosystems than the first-party ones, resulting in higher trust and fewer concerns towards the third-party ecosystems. We discuss the design and policy implications based on our results.

Figures (8)

• Figure 1: Successful booking confirmation from Gemini (Bard)
• Figure 2: Chatbot as a Key Player Drawing from P2. The data originated from the user and was transmitted to Gemini (Bard), from there it was further transferred to Google, as well as to an entity marked with a question mark, referred to by P2 as the "black box." He believed Gemini collected information and then transmitted it to other uncertain entities other than Google.
• Figure 3: Chatbot as a Key Player Drawing from P20. In P20's drawing, servers were depicted with two rectangles of different sizes, representing the dominant Gemini (Bard) and the subordinate Google. Gemini transferred data to hotels for booking, and it could also be seen that after multiple exchanges within the chatbot ecosystem, the data was transferred back to Gemini's server.
• Figure 4: GenAI as a Medium Drawing from P1. P1 believed Gemini (Bard) merely acted as a conduit for transmitting information to its parent company, Google. It was Google that actually generated choices for the user and shared information with other vendors.
• Figure 5: GenAI as a Representation Drawing from P19 (In the diagram, Gemini was used to denote both Gemini (Bard) and Google, with the depiction of Google being omitted. Initially, data was transferred from the user to Gemini. Following an internal data exchange between Gemini and Google, Gemini then acquired more specific information from the user to further the hotel booking process.)