A Deixis-Centered Approach for Documenting Remote Synchronous Communication around Data Visualizations

TL;DR

The paper tackles the problem that traditional meeting documentation fails to capture deixis—cursor-based referential gestures commonly used around data visualizations. It proposes a four-stage framework that collects audio, pointer gestures, and interaction provenance, uses an LLM to align gestures with utterances, and generates interactive notes that link transcripts and minutes to animated gesture annotations. Empirical studies show participants prefer these interactive notes to standard recordings or transcripts, and the work introduces a preliminary taxonomy of cursor-based deictic gestures. The contributions enable richer, replayable documentation of collaborative data analysis and can be integrated into existing remote-work visualization tools to improve traceability and review efficiency.

Abstract

Referential gestures, or as termed in linguistics, deixis, are an essential part of communication around data visualizations. Despite their importance, such gestures are often overlooked when documenting data analysis meetings. Transcripts, for instance, fail to capture gestures, and video recordings may not adequately capture or emphasize them. We introduce a novel method for documenting collaborative data meetings that treats deixis as a first-class citizen. Our proposed framework captures cursor-based gestural data along with audio and converts them into interactive documents. The framework leverages a large language model to identify word correspondences with gestures. These identified references are used to create context-based annotations in the resulting interactive document. We assess the effectiveness of our proposed method through a user study, finding that participants preferred our automated interactive documentation over recordings, transcripts, and manual note-taking. Furthermore, we derive a preliminary taxonomy of cursor-based deictic gestures from participant actions during the study. This taxonomy offers further opportunities for better utilizing cursor-based deixis in collaborative data analysis scenarios.

Figures (7)

• Figure 1: An overview of the collaborative interface, serving to enable collaborative visualization and data collections. It consists of (A) a gallery of meeting materials, (B) a collaborative board, and (C) room controls.
• Figure 2: An illustration of the operational mechanisms underpinning the interactive visualization demo within our collaborative interface. In this example, "Client 1" clicked on the node "Marius". This event is relayed to the server and then broadcast to all connected clients. Concurrently, the change of the current selected node and node positions are stored in the state file.
• Figure 3: Illustration of various scenarios for matching utterances with transient referential gestures. (A) One gesture matches with one sentence. (B) Several gestures match with one sentence. (C) One gesture matches with several sentences.
• Figure 4: Illustration of reference extraction: (a) Two imprecise gestures are matched with one sentence. (b) After reference extraction, gestures are sequentially associated with two separate phrases.
• Figure 5: An overview of the interactive notes, with: (A) Interactive text, comprising transcripts from audio and the LLM-generated meeting minutes, includes interactive text components based on the results of utterance matching and reference extraction. (B) Visual media from the meetings are presented with annotations based on parameters transmitted by the interactive text on the left. This operation can change the underlying visualization, add annotations, and alter interactive states.