Spatial Affordance-aware Interactable Subspace Allocation for Mixed Reality Telepresence

TL;DR

This work addresses MR telepresence across multiple remote spaces by introducing SA-ISA, a spatial affordance-aware interactable subspace allocation method. By separating perceivable space from context-specific interactable subspaces, SA-ISA enables multiple remote clients to access and interact with an AR host's space while maintaining personal space, using scene-graph–driven target selection, a context-aware objective function that blends semantic, geometric, and interaction terms, and a marker-based subspace extraction mechanism. The approach demonstrates superior performance in instantiating users and preserving wide mutual interactable areas across 900 realistic space composites, outperforming semantic total intersection and previous ISA baselines, especially as client numbers grow. The results highlight the method's potential to enhance multi-user MR telepresence in diverse environments, with implications for scalable remote collaboration and the explicit handling of spatial affordances and proxemics, albeit with noted limitations in real-time computation and vertical alignment that future work aims to address.

Abstract

To enable remote Virtual Reality (VR) and Augmented Reality (AR) clients to collaborate as if they were in the same space during Mixed Reality (MR) telepresence, it is essential to overcome spatial heterogeneity and generate a unified shared collaborative environment by integrating remote spaces into a target host space. Especially when multiple remote users connect, a large shared space is necessary for people to maintain their personal space while collaborating, but the existing simple intersection method leads to the creation of narrow shared spaces as the number of remote spaces increases. To robustly align to the host space even as the number of remote spaces increases, we propose a spatial affordance-aware interactable subspace allocation algorithm. The key concept of our approach is to consider the perceivable and interactable areas separately, where every user views the same mutual space, but each remote user has a different interactable subspace, considering their location and spatial affordance. We conducted an evaluation with 900 space combinations, varying the number of remote spaces as two, four, and six, and results show our method outperformed in securing wide interactable mutual space and instantiating users compared to the other spatial matching methods. Our work enables multiple clients from diverse remote locations to access the AR host's space, allowing them to interact directly with the table, wall, or floor by aligning their physical subspaces within a connected mutual space.

Figures (5)

• Figure 1: The overview of spatial matching for MR mutual space generation in a table-centric collaboration. A) Select a target object from each client's space corresponding to the host's target object with a scene graph, B) collaboration context-aware spatial matching, C) spatial affordance-aware user instantiation, and D) interactable subspace extraction and allocation.
• Figure 2: Top view of four host spaces (four meeting rooms) and ten client spaces (five homes and five offices) used for evaluation.
• Figure 3: The user instantiation success rate in each condition. A) H1-C2, B) H1-C4, and C) H1-C6.
• Figure 4: The mean of interactable space area and obstacle area in each condition. A) Mean total mutual space area in A-1) H1-C2, A-2) H1-C4, and A-3) H1-C6. B) Mean interactable area per client and C) mean obstacle area per client.
• Figure 5: The representative spatial matching results of our method (SA-Table, SA-Wall, and SA-Floor) and comparison groups (S-ISA and S-TI). A) H1-C2, B) H1-C4, and C) H1-C6. H1, H2, and H3 refer to the sequential order of instantiated host users. C1 from C6 refers to clients from each space. The gray areas refer to semantically unmatched areas, and the instantiation failed users were written outside the host's space.