Hey Dashboard!: Supporting Voice, Text, and Pointing Modalities in Dashboard Onboarding

TL;DR

This work addresses the onboarding bottleneck in complex dashboards by proposing Diana, a multimodal dashboard onboarding assistant that integrates voice, text, and pointing interactions within the dashboard interface. Diana leverages a GPT-powered agent, contextual visual highlights, and a radial help menu to provide situated explanations using actual dashboard metadata, without performing data queries. Through a two-phase qualitative study with novices and experts, the authors show that users rapidly adopt Diana and prefer multimodal interactions, with voice and visual highlights emerging as the most valued cues. The findings suggest practical design guidelines for multimodal onboarding tools that empower users to learn dashboards autonomously while maintaining guidance and context through integrated AI-based assistance.

Abstract

Visualization dashboards are regularly used for data exploration and analysis, but their complex interactions and interlinked views often require time-consuming onboarding sessions from dashboard authors. Preparing these onboarding materials is labor-intensive and requires manual updates when dashboards change. Recent advances in multimodal interaction powered by large language models (LLMs) provide ways to support self-guided onboarding. We present DIANA (Dashboard Interactive Assistant for Navigation and Analysis), a multimodal dashboard assistant that helps users for navigation and guided analysis through chat, audio, and mouse-based interactions. Users can choose any interaction modality or a combination of them to onboard themselves on the dashboard. Each modality highlights relevant dashboard features to support user orientation. Unlike typical LLM systems that rely solely on text-based chat, DIANA combines multiple modalities to provide explanations directly in the dashboard interface. We conducted a qualitative user study to understand the use of different modalities for different types of onboarding tasks and their complexities.

Figures (4)

• Figure 1: Multimodal interaction for dashboard onboarding using LLMs. Using the process model for dashboard onboarding proposed by Dhanoa et al. v_process_2022, we extend it with multimodal interactions and features, powered by LLMs for onboarding a user.
• Figure 2: Diana User Interface. The left side shows a visualization dashboard with Diana activated: (a) represents the lasso selection button, (b) push-to-talk for speech-based interaction, and (c) chat-based interaction. Currently, the lasso selection is activated and the y-axis of the visualization is highlighted, along with the radial menu which is opened for the y-axis information. The right side shows multimodal interaction with mouse-based lasso selection and chat-based interaction that infers the context of the conversation based on the selected visual.
• Figure 3: Participant demographics. Demographics of the formative study participants (A1-A4) and main study participants (P1-P6). Gender is encoded as male (M) and female (F). Visualization and dashboard experience is measured from novice to expert. AI interface experiences is noted for chat and voice interfaces.
• Figure 4: Modalities and features used during onboarding. Filled boxes show the use of a modality or a feature. Empty boxes indicate that it was not used. Boxes are colored by participants ID: P1 , P2 , P3 , P4 , P5 , and P6 .