Enriching Physical-Virtual Interaction in AR Gaming by Tracking Identical Real Objects

TL;DR

This paper tackles the challenge of tracking visually identical real objects in AR games using only partial observations from an AR headset. It introduces an optimization-based label-assignment framework, where object identities are resolved by solving an integer programming problem that minimizes a total cost composed of translation, rotation, and dimension terms, $C^{i\to j} = C_d^{i\to j}(w_t C_t^{i\to j} + w_r C_r^{i\to j})$, subject to assignment constraints, while a Voronoi-diagram–driven pruning strategy reduces the search space. The approach leverages pose estimation from RGB(-D) data (e.g., Objectron) and projects detections into the world to compute costs and perform identity assignment, enabling robust physical–virtual interactions without additional hardware. The method is validated through synthetic quantitative experiments and multiple qualitative AR scenarios, including a Farm-to-Table headset AR game, AR storytelling, and a gaming robot demonstration, showing practical applicability and potential for real-time, scalable AR experiences. Overall, the work offers a scalable, headset-centric solution to maintain coherent interactions in dynamic scenes with many identical objects, reducing the need for fixed cameras or markers and enabling more immersive AR gameplay and narratives.

Abstract

Augmented reality (AR) games, particularly those designed for headsets, have become increasingly prevalent with advancements in both hardware and software. However, the majority of AR games still rely on pre-scanned or static scenes, and interaction mechanisms are often limited to controllers or hand-tracking. Additionally, the presence of identical objects in AR games poses challenges for conventional object tracking techniques, which often struggle to differentiate between identical objects or necessitate the installation of fixed cameras for global object movement tracking. In response to these limitations, we present a novel approach to address the tracking of identical objects in an AR scene to enrich physical-virtual interaction. Our method leverages partial scene observations captured by an AR headset, utilizing the perspective and spatial data provided by this technology. Object identities within the scene are determined through the solution of a label assignment problem using integer programming. To enhance computational efficiency, we incorporate a Voronoi diagram-based pruning method into our approach. Our implementation of this approach in a farm-to-table AR game demonstrates its satisfactory performance and robustness. Furthermore, we showcase the versatility and practicality of our method through applications in AR storytelling and a simulated gaming robot. Our video demo is available at: https://youtu.be/rPGkLYuKvCQ.

Figures (17)

• Figure 1: Mobile AR game: Pokémon GO.
• Figure 2: Example headset AR games. Image courtesy of the game developers.
• Figure 3: An overview of our approach.
• Figure 4: Screenshots of the six scenes with different complexity levels. Note that although shown in approximately the same size for visual clarity, the layouts are scaled down by different amounts due to different original sizes.
• Figure 5: A demonstration of the threshold experiment.