VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas

TL;DR

VirtualNexus addresses the key limitation of 360-degree telepresence—the lack of depth and interactive physics—by uniting high-fidelity 360° video with spatially aligned 3D reconstructions, environment cutouts, and on-demand virtual replicas. It introduces four design concepts (spatial physicality, environment cutouts, ad-hoc virtual replicas via Instant-NGP NeRF, and spatially synchronized collaboration) and provides an end-to-end implementation with Unity, commodity hardware, and a fast replica pipeline. The system supports synchronized pointers/annotations and shared objects, plus an object-scanning workflow that generates NeRF-based replicas within minutes. A dyadic user study across three application scenarios demonstrates enhanced clarity, presence, and collaboration efficiency, suggesting that VirtualNexus meaningfully extends the interaction space and practical utility of 360° telepresence for remote work, education, and recreation.

Abstract

Asymmetric AR/VR collaboration systems bring a remote VR user to a local AR user's physical environment, allowing them to communicate and work within a shared virtual/physical space. Such systems often display the remote environment through 3D reconstructions or 360-degree videos. While 360-degree cameras stream an environment in higher quality, they lack spatial information, making them less interactable. We present VirtualNexus, an AR/VR collaboration system that enhances 360-degree video AR/VR collaboration with environment cutouts and virtual replicas. VR users can define cutouts of the remote environment to interact with as a world-in-miniature, and their interactions are synchronized to the local AR perspective. Furthermore, AR users can rapidly scan and share 3D virtual replicas of physical objects using neural rendering. We demonstrated our system's utility through 3 example applications and evaluated our system in a dyadic usability test. VirtualNexus extends the interaction space of 360-degree telepresence systems, offering improved physical presence, versatility, and clarity in interactions.

Figures (9)

• Figure 1: The system has four major components: VR side, AR side, video server, and virtual replica server. The VR side receives 360$\degree$ video from the AR side via the video server. The AR side shares synced objects and annotations with the VR side and sends scanned RGBD images to the virtual replica server, which creates virtual replicas and sends it to both the AR and VR sides.
• Figure 2: Spatial-accurate Alignment of 3D Reconstruction with the 360$\degree$ Video. The edges of the spatial mesh are only coloured in red here for demonstrative purposes.
• Figure 3: Environment Cutouts: In (a), the VR user defines a cutout of the whiteboard through 4 raycasted points. In (b), annotations on active cutouts are synced to the original location. In (c), users can cutout 3D space additional to 2D surfaces.
• Figure 4: Intermediate results for rapid virtual replica creation: (a) original RGB image (b) background removed image (c) volume rendering by Instant-NGP (d) mesh object created from cube-marching (e) Voxelized and smoothed final object.
• Figure 5: The collaborative prototyping scenario: The users collaboratively create a story scene. In (a) and (b), both VR and AR users use shared virtual objects for the scene. Subimages (c) and (d) show the completed scene from the VR and AR perspectives.