Glass Chirolytics: Reciprocal Compositing and Shared Gestural Control for Face-to-Face Collaborative Visualization at a Distance

TL;DR

This paper employs a reciprocal compositing of visualization and interface widgets over the mirrored video of one's conversation partner, suggestive of a pane of glass, in which both parties can simultaneously manipulate composited elements via bimanual gestures.

Abstract

Videoconference conversations about data often entail screen sharing visualization artifacts, in which nonverbal communication goes largely ignored. Beyond presentation use cases, conversations supported by visualization also arise in collaborative decision making, technical interviews, and tutoring: use cases that benefit from participants being able to see one another as they exchange questions about the data. In this paper, we employ a reciprocal compositing of visualization and interface widgets over the mirrored video of one's conversation partner, suggestive of a pane of glass, in which both parties can simultaneously manipulate composited elements via bimanual gestures. We demonstrate our approach with implementations of several visualization interfaces spanning the aforementioned use cases, and we evaluate our approach in a study (N = 16) comparing it to videoconferencing while using a mouse to interact with a collaborative web application. Our findings suggest that our approach promotes feelings of presence and mutual awareness of analytical intent.

Figures (8)

• Figure 1: In this illustrated instance of the Glass Chirolytics approach, a remote collaborator appears composited behind a synchronized node-link diagram. The collaborator on the left points with their right hand to select one node while the collaborator on the right selects multiple nodes with their left hand; both can see a reflection of their hands as a green skeletal mesh composited in their local foreground. Lastly, the combination of their gestures triggers the reveal of highlighted links between the set of selected nodes.
• Figure 2: Our gestural vocabulary includes: (a) pointing to ephemerally select an individual element; (b) point-and-tap to persist an individual selection; (c) spread to coarsely select multiple elements (ephemeral unless held for one second); (d) pinch-and-move to reconfigure the positions of elements; (e) grab-and-move panning; and (f) separate-or-join zooming. Recognition of the latter two gestures follows a one-second delay communicated with an ephemeral green indicator icon appearing at the base of the wrists.
• Figure 3: Exploratory analysis scenarios: in Scene 1, David spreads their hand to select a group of establishment legislators on the right, revealing their voting patterns. In Scene 2, David spreads their right hand to select states in the American Southeast, revealing the largest sources of incoming domestic migration to those states, while Abhi filters the data with their left hand to reflect the 2020s.
• Figure 4: Tutoring scenarios: In Scene 1, Abhi points to different nodes with both hands to reveal the shortest path between them, while David moves the path's intermediary node a new position via pinch-and-move. In Scene 2, Abhi spreads their right hand to select points in the bottom-right plot of a SPLOM, revealing the selected data points' positions in the other plots.
• Figure 5: Technical interviewing scenarios: In Scene 1, David repositions two elements in a system design diagram. In Scene 2, David points at two nodes in a Sankey energy diagram to highlight the flow of natural gas to the electric grid.