Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Digital Twin-based 3D Map Management for Edge-assisted Device Pose Tracking in Mobile AR

Conghao Zhou, Jie Gao, Mushu Li, Nan Cheng, Xuemin Shen, Weihua Zhuang

TL;DR

This work tackles the challenge of resource-constrained mobile AR pose tracking by enabling edge-assisted 3D map management that adapts to time-varying uplink data rates. It introduces a digital twin (DT) framework, notably a User DT (UDT), to model uplink dynamics and supply latent features for a Bayes-adaptive MDP formulation, solved via model-based DRL with blended real and artificial experiences. The proposed adaptive map-management (AMM) algorithm prioritizes camera frames for uploading and map updating to minimize pose estimation uncertainty, outperforming several baselines. Numerical results validate the DT’s accuracy in capturing uplink dynamics and demonstrate that the DT-enabled MBRL approach reduces pose uncertainty under dynamic network and pose variations, offering a practical path toward robust edge-assisted MAR in 6G-era networks.

Abstract

Edge-device collaboration has the potential to facilitate compute-intensive device pose tracking for resource-constrained mobile augmented reality (MAR) devices. In this paper, we devise a 3D map management scheme for edge-assisted MAR, wherein an edge server constructs and updates a 3D map of the physical environment by using the camera frames uploaded from an MAR device, to support local device pose tracking. Our objective is to minimize the uncertainty of device pose tracking by periodically selecting a proper set of uploaded camera frames and updating the 3D map. To cope with the dynamics of the uplink data rate and the user's pose, we formulate a Bayes-adaptive Markov decision process problem and propose a digital twin (DT)-based approach to solve the problem. First, a DT is designed as a data model to capture the time-varying uplink data rate, thereby supporting 3D map management. Second, utilizing extensive generated data provided by the DT, a model-based reinforcement learning algorithm is developed to manage the 3D map while adapting to these dynamics. Numerical results demonstrate that the designed DT outperforms Markov models in accurately capturing the time-varying uplink data rate, and our devised DT-based 3D map management scheme surpasses benchmark schemes in reducing device pose tracking uncertainty.

Digital Twin-based 3D Map Management for Edge-assisted Device Pose Tracking in Mobile AR

TL;DR

This work tackles the challenge of resource-constrained mobile AR pose tracking by enabling edge-assisted 3D map management that adapts to time-varying uplink data rates. It introduces a digital twin (DT) framework, notably a User DT (UDT), to model uplink dynamics and supply latent features for a Bayes-adaptive MDP formulation, solved via model-based DRL with blended real and artificial experiences. The proposed adaptive map-management (AMM) algorithm prioritizes camera frames for uploading and map updating to minimize pose estimation uncertainty, outperforming several baselines. Numerical results validate the DT’s accuracy in capturing uplink dynamics and demonstrate that the DT-enabled MBRL approach reduces pose uncertainty under dynamic network and pose variations, offering a practical path toward robust edge-assisted MAR in 6G-era networks.

Abstract

Edge-device collaboration has the potential to facilitate compute-intensive device pose tracking for resource-constrained mobile augmented reality (MAR) devices. In this paper, we devise a 3D map management scheme for edge-assisted MAR, wherein an edge server constructs and updates a 3D map of the physical environment by using the camera frames uploaded from an MAR device, to support local device pose tracking. Our objective is to minimize the uncertainty of device pose tracking by periodically selecting a proper set of uploaded camera frames and updating the 3D map. To cope with the dynamics of the uplink data rate and the user's pose, we formulate a Bayes-adaptive Markov decision process problem and propose a digital twin (DT)-based approach to solve the problem. First, a DT is designed as a data model to capture the time-varying uplink data rate, thereby supporting 3D map management. Second, utilizing extensive generated data provided by the DT, a model-based reinforcement learning algorithm is developed to manage the 3D map while adapting to these dynamics. Numerical results demonstrate that the designed DT outperforms Markov models in accurately capturing the time-varying uplink data rate, and our devised DT-based 3D map management scheme surpasses benchmark schemes in reducing device pose tracking uncertainty.
Paper Structure (31 sections, 2 theorems, 25 equations, 10 figures, 1 table, 2 algorithms)

This paper contains 31 sections, 2 theorems, 25 equations, 10 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Cloud/Edge-assisted Device Pose Tracking
  4. Computing Task Offloading and Scheduling
  5. System Model
  6. Considered Scenario
  7. Workflow of Edge-assisted Device Pose Tracking
  8. 3D Map Model
  9. Camera Frame Uploading
  10. 3D Map Update
  11. The impact of resource limitation for solving a 3D map optimization problem
  12. 3D map update
  13. Pose Estimation Uncertainty
  14. Problem Formulation & Transformation
  15. Problem Formulation
  16. ...and 16 more sections

Key Result

Lemma 1

Given a connected 3D map $\mathcal{G}=(\mathcal{V}, \mathcal{E})$, the pose estimation uncertainty $u(\mathcal{G})$ monotonously decreases with the value of $|\mathcal{V}|$ when $\det(\boldsymbol{\Pi}) \ge 1$.

Figures (10)

  • Figure 1: The considered scenario of edge-assisted MAR.
  • Figure 2: The timeline of 3D map management.
  • Figure 3: An illustration of the 3D map model.
  • Figure 4: The workflow of the designed UDT and UDT-based 3D map management.
  • Figure 5: The estimated and the actual values of the state transition matrix of a $4$-state Markov chain within one time interval.
  • ...and 5 more figures

Theorems & Definitions (5)

  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • proof