Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive 3D UI Placement in Mixed Reality Using Deep Reinforcement Learning

Feiyu Lu, Mengyu Chen, Hsiang Hsu, Pranav Deshpande, Cheng Yao Wang, Blair MacIntyre

TL;DR

This work investigates using deep reinforcement learning to continuously place a 3D MR UI in mobile, dynamic environments. By training a PPO-based agent in a physics-driven simulation with a rich observation set, the approach aims to maximize long-term user utility through adaptive content placement. Preliminary evaluations across multiple indoor scenes show the model can maintain high visibility and reachability with limited performance drops under dynamic obstacles, suggesting potential for generalization and real-world applicability, while also highlighting stability and overfitting challenges. The study outlines multiple avenues for future work, including richer reward signals, multi-user scenarios, human-in-the-loop feedback, and model-based RL approaches to further improve robustness and personalization in MR UIs.

Abstract

Mixed Reality (MR) could assist users' tasks by continuously integrating virtual content with their view of the physical environment. However, where and how to place these content to best support the users has been a challenging problem due to the dynamic nature of MR experiences. In contrast to prior work that investigates optimization-based methods, we are exploring how reinforcement learning (RL) could assist with continuous 3D content placement that is aware of users' poses and their surrounding environments. Through an initial exploration and preliminary evaluation, our results demonstrate the potential of RL to position content that maximizes the reward for users on the go. We further identify future directions for research that could harness the power of RL for personalized and optimized UI and content placement in MR.

Adaptive 3D UI Placement in Mixed Reality Using Deep Reinforcement Learning

TL;DR

This work investigates using deep reinforcement learning to continuously place a 3D MR UI in mobile, dynamic environments. By training a PPO-based agent in a physics-driven simulation with a rich observation set, the approach aims to maximize long-term user utility through adaptive content placement. Preliminary evaluations across multiple indoor scenes show the model can maintain high visibility and reachability with limited performance drops under dynamic obstacles, suggesting potential for generalization and real-world applicability, while also highlighting stability and overfitting challenges. The study outlines multiple avenues for future work, including richer reward signals, multi-user scenarios, human-in-the-loop feedback, and model-based RL approaches to further improve robustness and personalization in MR UIs.

Abstract

Mixed Reality (MR) could assist users' tasks by continuously integrating virtual content with their view of the physical environment. However, where and how to place these content to best support the users has been a challenging problem due to the dynamic nature of MR experiences. In contrast to prior work that investigates optimization-based methods, we are exploring how reinforcement learning (RL) could assist with continuous 3D content placement that is aware of users' poses and their surrounding environments. Through an initial exploration and preliminary evaluation, our results demonstrate the potential of RL to position content that maximizes the reward for users on the go. We further identify future directions for research that could harness the power of RL for personalized and optimized UI and content placement in MR.
Paper Structure (17 sections, 1 equation, 2 figures, 1 table)

This paper contains 17 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adaptive user interfaces in mixed reality
  4. Reinforcement learning for interface adaptations
  5. Problem Formulation
  6. Environment & User Simulation
  7. RL Agent
  8. Observations
  9. Action Space
  10. Reward Design
  11. Training Configuration & Results
  12. Preliminary Evaluation
  13. Setup
  14. Results
  15. Future Opportunities
  16. ...and 2 more sections

Figures (2)

  • Figure 1: (a) Simulating user movement during training by dynamically interpolating an avatar along a 3D Cartesian grid facing a random direction every few seconds; (b) the accumulative reward per episode (y-axis) in relation to the number of total steps (x-axis); (c) the histogram of the reward distribution (x-axis) in relation to the total number of steps (y-axis).
  • Figure 2: (a) A snapshot of the training in which a UI driven by RL gauges its distances to surroundings and simulated user poses; (b) a snapshot of the dynamic obstacle test, in which the content (blue square) maintains visibility, reachability and avoiding collision with a presence of a randomly moving obstacle (red cube); (c) a snapshot of the placement result running our trained RL model.