VizCopilot: Fostering Appropriate Reliance on Enterprise Chatbots with Context Visualization

TL;DR

Enterprise chatbots often produce irrelevantly right responses due to context misalignment between retrieved data and user intent. VizCopilot adds a visualization panel that visualizes topic-structured context and provides group-level control, enabling end-users to oversee and modify retrieved data with low cognitive overhead. In a design-through-design study with 14 experienced users, the visualization improved context alignment, transparency of the generative process, and prompting strategies, though verification support and trust in AI-generated summaries remain limited. The work contributes a functional prototype, an extended treemap interface with progressive disclosure, and design guidelines for visualization-enhanced conversational systems, highlighting directions toward personalization, proactivity, and sustainable human–AI collaboration in enterprise knowledge work.

Abstract

Enterprise chatbots show promise in supporting knowledge workers in information synthesis tasks by retrieving context from large, heterogeneous databases before generating answers. However, when the retrieved context misaligns with user intentions, the chatbot often produces "irrelevantly right" responses that provide little value. In this work, we introduce VizCopilot, a prototype that incorporates visualization techniques to actively involve end-users in context alignment. By combining topic modeling with document visualization, VizCopilot enables human oversight and modification of retrieved context while keeping cognitive overhead manageable. We used VizCopilot as a design probe in a Research-through-Design study to evaluate the role of visualization in context alignment and to surface future design opportunities. Our findings show that visualization not only helps users detect and correct misaligned context but also encourages them to adapt their prompting strategies, enabling the system to retrieve more relevant context from the outset. At the same time, the study reveals limitations in verification support regarding close-reading and trust in AI summaries. We outline future directions for visualization-enhanced chatbots, focusing on personalization, proactivity, and sustainable human-AI collaboration.

Figures (4)

• Figure 1: Overview of VizCopilot before entering a prompt. (a) The visualization panel shows topic structures of the context in a treemap-based visualization. (b) The Copilot chat panel allows for typical conversational interactions with an extension of the data context panel. (c) Each cell can be switched between the canvas view (default) and the file view, which allows for direct inspection of file content. (d)Each cell in the treemap can be expanded to allocate more space for visual clarity.
• Figure 2: Interactions supported by VizCopilot. (a) Users can enter their prompt in the chat panel to initiate a conversation. (b) VizCopilot uses the prompt to retrieve context data, highlight it on the visualization, and automatically summarize it according to the subtopics. (c) Users can drill down to individual subtopics and inspect the file contents. (d) Users can use the drag-and-drop feature to modify context for follow-up prompts.
• Figure 3: The highlight feature allows users to quickly check the alignment of retrieved context. When the user enters "Summarize everything related to marketing", the topic for marketing is expected to be highlighted, while the other two topics call for manual checks.
• Figure 4: An example of Copilot misinterpreting the context. Copilot consistently mistakens different employees with the same name as the same person, despite the direct question. Most participants in the user study were able to identify such an error by inspecting the visualization panel in the file view.