Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PointAloud: An Interaction Suite for AI-Supported Pointer-Centric Think-Aloud Computing

Frederic Gmeiner, John Thompson, George Fitzmaurice, Justin Matejka

TL;DR

PointAloud addresses the challenge of capturing and leveraging designers' evolving thoughts by tightly integrating Think-Aloud Computing with pointer-centric, ambient feedback and AI-driven assistance within CAD workflows. The system introduces TalkPointer, TalkNotes, and TalkTips to capture verbalizations, anchor them to spatial design context, and provide proactive guidance, while TalkExplorer and TalkReminders support retrieval and recall. In a within-subject study with 12 professionals, PointAloud improved perceived task relevance, process awareness, and recapability relative to a text-based baseline, though it produced mixed effects on speech quantity and prompt usefulness. The work delivers design principles for incentive-driven verbalization, ambient pointer displays, and process-aware human–AI co-creation, offering a transferable approach for documenting design processes across tools and domains and advancing AI-supported design workflows.

Abstract

Think-Aloud Computing, a method for capturing users' verbalized thoughts during software tasks, allows eliciting rich contextual insights into evolving intentions, struggles, and decision-making processes of users in real-time. However, existing approaches face practical challenges: users often lack awareness of what is captured by the system, are not effectively encouraged to speak, and miss or are interrupted by system feedback. Additionally, thinking aloud should feel worthwhile for users due to the gained contextual AI assistance. To better support and harness Think-Aloud Computing, we introduce PointAloud, a suite of novel AI-driven pointer-centric interactions for in-the-moment verbalization encouragement, low-distraction system feedback, and contextually rich work process documentation alongside proactive AI assistance. Our user study with 12 participants provides insights into the value of pointer-centric think-aloud computing for work process documentation and human-AI co-creation. We conclude by discussing the broader implications of our findings and design considerations for pointer-centric and AI-supported Think-Aloud Computing workflows.

PointAloud: An Interaction Suite for AI-Supported Pointer-Centric Think-Aloud Computing

TL;DR

PointAloud addresses the challenge of capturing and leveraging designers' evolving thoughts by tightly integrating Think-Aloud Computing with pointer-centric, ambient feedback and AI-driven assistance within CAD workflows. The system introduces TalkPointer, TalkNotes, and TalkTips to capture verbalizations, anchor them to spatial design context, and provide proactive guidance, while TalkExplorer and TalkReminders support retrieval and recall. In a within-subject study with 12 professionals, PointAloud improved perceived task relevance, process awareness, and recapability relative to a text-based baseline, though it produced mixed effects on speech quantity and prompt usefulness. The work delivers design principles for incentive-driven verbalization, ambient pointer displays, and process-aware human–AI co-creation, offering a transferable approach for documenting design processes across tools and domains and advancing AI-supported design workflows.

Abstract

Think-Aloud Computing, a method for capturing users' verbalized thoughts during software tasks, allows eliciting rich contextual insights into evolving intentions, struggles, and decision-making processes of users in real-time. However, existing approaches face practical challenges: users often lack awareness of what is captured by the system, are not effectively encouraged to speak, and miss or are interrupted by system feedback. Additionally, thinking aloud should feel worthwhile for users due to the gained contextual AI assistance. To better support and harness Think-Aloud Computing, we introduce PointAloud, a suite of novel AI-driven pointer-centric interactions for in-the-moment verbalization encouragement, low-distraction system feedback, and contextually rich work process documentation alongside proactive AI assistance. Our user study with 12 participants provides insights into the value of pointer-centric think-aloud computing for work process documentation and human-AI co-creation. We conclude by discussing the broader implications of our findings and design considerations for pointer-centric and AI-supported Think-Aloud Computing workflows.
Paper Structure (57 sections, 31 figures, 3 tables)

This paper contains 57 sections, 31 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pointer-Centric Display Techniques
  4. Context-Aware Support Systems
  5. Workflow Capture, Documentation, and Retrieval Systems
  6. Exploring Interaction Design Principles for Supporting Think-Aloud Computing
  7. Design Principles
  8. PointAloud System: A Pointer-Centric AI-supported Think-Aloud Computing System
  9. Example Use Case Scenarios
  10. 2D Use Case: Annotating the floor plan
  11. 3D Use Case: Preparing for a client meeting
  12. Interface Features
  13. System Interface Overview
  14. TalkPointer: Dynamic Feedback Anchored to the Cursor (DP1, DP4)
  15. TalkNotes: Contextualized Representations of Design Moments (DP2, DP3, DP4)
  16. ...and 42 more sections

Figures (31)

  • Figure 1: Screenshot of the PointAloud system: (A) Main canvas with activated 2D view containing sketches on a floor plan with an unfolded TalkNote card and TalkPointer next to the mouse cursor; (B) TalkExplorer sidebar for browsing and filtering of captured TalkNotes; (C) Menu bar with controls for starting/stopping transcription and switching between 2D and (D) 3D view.
  • Figure 2: Pointer-adjacent TalkPointer display comprising TalkTip, TalkText, and TalkViz
  • Figure 3:
  • Figure 4:
  • Figure 5:
  • ...and 26 more figures