SpatialTouch: Exploring Spatial Data Visualizations in Cross-reality

TL;DR

SpatialTouch introduces a cross-reality system that unites a monoscopic 2D touch surface with a stereoscopic AR space to explore spatial data. Through an elicitation study and domain-specific deployments in astronomy, molecular biology, and medical imaging, the work develops a design space for CR visualization and interaction, including novel transition techniques that maintain spatial awareness across 2D and 3D representations. Domain and VR/AR/MR expert evaluations validate the approach, revealing strong usability, intuitive cross-space interactions, and valuable insights for data transformations and representation control. The study demonstrates that enabling users to choose where data is displayed and how it is interacted with—alongside seamless visualization transitions—significantly enhances data understanding and task performance in cross-reality visualization contexts.

Abstract

We propose and study a novel cross-reality environment that seamlessly integrates a monoscopic 2D surface (an interactive screen with touch and pen input) with a stereoscopic 3D space (an augmented reality HMD) to jointly host spatial data visualizations. This innovative approach combines the best of two conventional methods of displaying and manipulating spatial 3D data, enabling users to fluidly explore diverse visual forms using tailored interaction techniques. Providing such effective 3D data exploration techniques is pivotal for conveying its intricate spatial structures -- often at multiple spatial or semantic scales -- across various application domains and requiring diverse visual representations for effective visualization. To understand user reactions to our new environment, we began with an elicitation user study, in which we captured their responses and interactions. We observed that users adapted their interaction approaches based on perceived visual representations, with natural transitions in spatial awareness and actions while navigating across the physical surface. Our findings then informed the development of a design space for spatial data exploration in cross-reality. We thus developed cross-reality environments tailored to three distinct domains: for 3D molecular structure data, for 3D point cloud data, and for 3D anatomical data. In particular, we designed interaction techniques that account for the inherent features of interactions in both spaces, facilitating various forms of interaction, including mid-air gestures, touch interactions, pen interactions, and combinations thereof, to enhance the users' sense of presence and engagement. We assessed the usability of our environment with biologists, focusing on its use for domain research. In addition, we evaluated our interaction transition designs with virtual and mixed-reality experts to gather further insights.

Figures (19)

• Figure 1: SpatialTouch's placement within the reality-virtuality continuum.
• Figure 2: The configurations of the AR camera and the Surface camera $\mathbf{c}$ are depicted, illustrating how the virtual content below the Surface (marked by a blue dotted line) is projected onto the Surface (a red solid line). The HoloLens view shows what users perceive in SpatialTouch.
• Figure 3: Design space for interaction techniques for two visualization tasks: data manipulation and selection. Red: interactions on 2D surface; Blue: interactions in 3D space. Below, Across, and Above: positions of the target data/location. (a), (d), (f): move data above the surface for interaction. (b) (c) and (e): interaction transitions across both spaces.
• Figure 4: Point cloud data selections: (a) BrushWYP, (b) BrushLasso.
• Figure 5: Proteins visualized with UnityMoldoutreligne:2014:unitymol. PDB ID: (a) https://doi.org/10.2210/pdb4FPQ/pdb, (b) https://doi.org/10.2210/pdb8RFE/pdb. Users can "grab" the 3D visualization directly to check the local regions.