Visualization for Human-Centered AI Tools

TL;DR

This work first interviewed HCI, AI, and Visualization experts to define the characteristics of HCAI tools, then presents several examples of HCAI tools using visualization and uses the examples to extract guidelines on how interactive visualization can support future HCAI tools.

Abstract

Human-centered AI (HCAI) puts the user in the driver's seat of so-called human-centered AI-infused tools (HCAI tools): interactive software tools that amplify, augment, empower, and enhance human performance using AI models. We discuss how interactive visualization can be a key enabling technology for creating such human-centered AI tools. To validate our approach, we first interviewed HCI, AI, and Visualization experts to define the characteristics of HCAI tools. We then present several examples of HCAI tools using visualization and use the examples to extract guidelines on how interactive visualization can support future HCAI tools.

Figures (5)

• Figure 1: Visualization-enabled HCAI tools. An interactive loop involving a human user and an AI model facilitated by visual interfaces.
• Figure 2: TimeFork. Line charts showing six stocks at a specific point in time. The analyst can make a prediction for one or more stocks by directly selecting (finger pointer) how the selected stock will change over time (dark blue line). In response, the temporal prediction model will show predictions for other visible stocks (orange line). Selecting a prediction advances the time to that position.
• Figure 3: The HaLLMark system. (A) Text editor for reading and editing. The system highlights text written (orange) and influenced (green) by the LLM. (B) Prompting interface for an LLM (e.g., GPT-4). The user can see the prompts and AI responses for the current session. (C) Summary statistics show the number of prompts and percentage of user-written text and AI assistance. Below is a timeline of a user's writing actions (grey rectangles) and interaction with the AI (purple and blue rectangles). It can be used to see prompts and linked text in the editor.
• Figure 4: Asking what-if questions in Outcome-Explorer. The directed acyclic graph shows causal relations between variables that determine median housing prices in a neighborhood. A user can create a user profile by moving the circular knobs in the nodes (variables). A user can keep one profile (green) fixed and change the other profile (orange) to ask what-if questions. The blue arrows indicate the changes in the orange profile. Note that property tax is set to 300 by the user. As a result, changing its parent nodes will not affect property tax. The other variables are estimated from their parents.
• Figure 5: Analyzing UX research data in uxSense. The uxSense interface is set up like a video editing interface in a web browser, but displays preprocessed data streams on a common timeline (bottom half of the screen) that is synchronized to the video playback (upper left) and transcript (upper center). The temporal visualizations on the timeline shows multiple different metrics derived from specialized AI models.