InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions

TL;DR

InterChat introduces a multimodal generative visual analytics system that combines natural language input with direct manipulation to enable precise expression of analytical intents and interactive visualization generation. It defines an explicit design space (intent, interaction, instruction) and uses a three-agent architecture to infer intents and generate D3.js visualizations via structured prompts and chain-of-thought reasoning. Through two real-world scenarios, a user study with ten participants, and industrial expert feedback, the work demonstrates improved accuracy and efficiency for complex, multi-step VA tasks, while highlighting challenges in latency, correctness, and data access. The study suggests that multimodal interactions can enhance interpretability, trust, and collaborative data analysis, laying the groundwork for future expansions including additional modalities and computational analytics capabilities.

Abstract

The rise of Large Language Models (LLMs) and generative visual analytics systems has transformed data-driven insights, yet significant challenges persist in accurately interpreting users' analytical and interaction intents. While language inputs offer flexibility, they often lack precision, making the expression of complex intents inefficient, error-prone, and time-intensive. To address these limitations, we investigate the design space of multimodal interactions for generative visual analytics through a literature review and pilot brainstorming sessions. Building on these insights, we introduce a highly extensible workflow that integrates multiple LLM agents for intent inference and visualization generation. We develop InterChat, a generative visual analytics system that combines direct manipulation of visual elements with natural language inputs. This integration enables precise intent communication and supports progressive, visually driven exploratory data analyses. By employing effective prompt engineering, and contextual interaction linking, alongside intuitive visualization and interaction designs, InterChat bridges the gap between user interactions and LLM-driven visualizations, enhancing both interpretability and usability. Extensive evaluations, including two usage scenarios, a user study, and expert feedback, demonstrate the effectiveness of InterChat. Results show significant improvements in the accuracy and efficiency of handling complex visual analytics tasks, highlighting the potential of multimodal interactions to redefine user engagement and analytical depth in generative visual analytics.

Figures (7)

• Figure 1: The design space of InterChat. The intent space defines users' goals within the generative VA system, with low-level interaction intents expressed via two modalities in the interaction space: Direct Manipulation and Natural Language Input. User's interaction, either through direct manipulation or natural language input (Interaction Space), are synthesized into structured prompts for LLM agents (Instruction Space, detailed in Section \ref{['sec:prompt_engineering']}), enabling interaction coordination and precise analytical intent expression.
• Figure 2: The system workflow. We use a multi-agent LLM architecture with two agents ($A_\mathrm{des}$, $A_\mathrm{link}$) for intent inference and one ($A_\mathrm{vis}$) for visualization generation. With our response processing module, users interact with visualizations to express analytical intents and inspect inferred intents through visual connections.
• Figure 3: The user interface of InterChat system, showing energy consumption data from a steel manufacturing company. DM1 and DM2 refer to two Direct Manipulation operations user performed on the visualization, where DM1 involves selecting a temporal range using the Box Selection tool and DM2 involves conveying a chart reconfiguration intent of moving the legend to the top-right corner.
• Figure 4: Prompt structure for visualization generation. Examples shown here uses Netflix stock price data. We only display the first few lines of the prompts. Full content is available in Appendix I.
• Figure 5: Usage scenario I: Netflix stock price data exploration and trend analysis. The user starts with a line chart (a), instruct the system to create Bollinger bands (c), uses free drawing tool to search for a specific trend (b), and employ box selection to select a specific time period (d). Our data binding mechanism allows users to inspect corresponding data items by clicking the chart elements (e).