BundleFit: Display and See-Through Models for Augmented Reality Head-Mounted Displays

TL;DR

BundleFit addresses the challenge of accurate, efficient AR HMD display and see-through modeling by treating optics as black boxes and fitting ray bundles on a fiber-bundle base space. The method leverages planar viewpoint sampling to build forward and backward display and see-through models, with intermediate polynomial fits, and integrates these into standard rendering pipelines. Validation via simulations and on-device experiments demonstrates improved accuracy and robustness over traditional white-box or volumetric black-box approaches, while maintaining hardware-agnostic applicability. This approach enables more reliable world-locked AR rendering and supports practical deployment in commercial AR headsets.

Abstract

The head-mounted display is a vital component of augmented reality, incorporating optics with complex display and see-through optical behavior. Computationally modeling these optical behaviors requires meeting three key criteria: accuracy, efficiency, and accessibility. In recent years, various approaches have been proposed to model display and see-through optics, which can broadly be classified into black-box and white-box models. However, both categories face significant limitations that hinder their adoption in commercial applications. To overcome these challenges, we leveraged prior knowledge of ray bundle properties outside the optical hardware and proposed a novel bundle-fit-based model. In this approach, the ray paths within the optics are treated as a black box, while a lightweight optimization problem is solved to fit the ray bundle outside the optics. This method effectively addresses the accuracy issues of black-box models and the accessibility challenges of white-box models. Although our model involves runtime optimization, this is typically not a concern, as it can use the solution from a previous query to initialize the optimization for the current query. We evaluated the performance of our proposed method through both simulations and experiments on real hardware, demonstrating its effectiveness.

Figures (10)

• Figure 1: Real example figure for World Locked Rendering.
• Figure 2: Explaination figure for World Locked Rendering.
• Figure 3: Pupil swim for display model.
• Figure 4: Volume sampling approach vs plane sampling.
• Figure 5: Volume sampling approach failure.