Exploring Spatial Hybrid User Interface for Visual Sensemaking

TL;DR

The paper addresses the limitations of desktop PCs and VR for visual sensemaking by proposing a spatial hybrid PC+VR system that blends a movable simulated PC in VR with synchronized cross-device states and hand-gesture control ($N=18$ participants in the main study). The authors outline five design requirements, implement an iterative prototype, and evaluate it against PC-only and VR-only conditions, finding that PC+VR is preferred—especially for interaction—without degrading task performance and with reduced physical demand relative to VR alone. Key contributions include a design framework for spatial hybrids, a functional PC+VR prototype, and empirical evidence that hybrid usage can enhance sensemaking workflows. The work demonstrates a viable path toward fatigue-resistant, space-efficient visual analytics by leveraging immersive space for overview and desktop precision for detailed work.

Abstract

We built a spatial hybrid system that combines a personal computer (PC) and virtual reality (VR) for visual sensemaking, addressing limitations in both environments. Although VR offers immense potential for interactive data visualization (e.g., large display space and spatial navigation), it can also present challenges such as imprecise interactions and user fatigue. At the same time, a PC offers precise and familiar interactions but has limited display space and interaction modality. Therefore, we iteratively designed a spatial hybrid system (PC+VR) to complement these two environments by enabling seamless switching between PC and VR environments. To evaluate the system's effectiveness and user experience, we compared it to using a single computing environment (i.e., PC-only and VR-only). Our study results (N=18) showed that spatial PC+VR could combine the benefits of both devices to outperform user preference for VR-only without a negative impact on performance from device switching overhead. Finally, we discussed future design implications.

Figures (14)

• Figure 1: The figure shows the PC, our proposed PC+VR hybrid system, and VR in reality–virtuality continuum milgram1995augmented. As the resulting environment is mainly virtual, it leans towards VR and falls under Augmented Virtuality.
• Figure 2: A demonstration of spatial hybrid systems for visual problem-solving: users can read documents and build a node-link diagram in VR. They interact with a digitally rendered flat screen on a physically movable table, using a mouse and keyboard to input text annotations. Notably, users do not need to put on and off the VR headset to switch between environments.
• Figure 3: Demonstrations of interfaces of PC-only. (a) document view, (b) graph view, (c) timeline view, and (d) minimap view.
• Figure 4: Demonstrations of interfaces of (a, b) VR-only and (c) PC+VR (a.k.a. hybrid). (a) document view and graph view, (b) timeline view, (c) tracked simulated PC in VR view.
• Figure 5: The figure shows different hand gestures implemented for graph (a-f) and timeline (g) manipulations. Red nodes indicate that the nodes will be deleted. Green nodes indicate that the nodes will be created. Yellow nodes indicate that the nodes are selected.