Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alignment of the Virtual Scene to the Tracking Space of a Mixed Reality Head-Mounted Display

Ehsan Azimi, Long Qian, Nassir Navab, Peter Kazanzides

TL;DR

This paper proposes a calibration procedure to properly align the coordinate system of a 3D virtual scene that the user sees with that of the tracker, and performs experiments with Microsoft HoloLens, taking advantage of its self localization and spatial mapping capabilities to eliminate the requirement for line of sight from the HMD to the object or external tracker.

Abstract

With the mounting global interest for optical see-through head-mounted displays (OST-HMDs) across medical, industrial and entertainment settings, many systems with different capabilities are rapidly entering the market. Despite such variety, they all require display calibration to create a proper mixed reality environment. With the aid of tracking systems, it is possible to register rendered graphics with tracked objects in the real world. We propose a calibration procedure to properly align the coordinate system of a 3D virtual scene that the user sees with that of the tracker. Our method takes a blackbox approach towards the HMD calibration, where the tracker's data is its input and the 3D coordinates of a virtual object in the observer's eye is the output; the objective is thus to find the 3D projection that aligns the virtual content with its real counterpart. In addition, a faster and more intuitive version of this calibration is introduced in which the user simultaneously aligns multiple points of a single virtual 3D object with its real counterpart; this reduces the number of required repetitions in the alignment from 20 to only 4, which leads to a much easier calibration task for the user. In this paper, both internal (HMD camera) and external tracking systems are studied. We perform experiments with Microsoft HoloLens, taking advantage of its self localization and spatial mapping capabilities to eliminate the requirement for line of sight from the HMD to the object or external tracker. The experimental results indicate an accuracy of up to 4 mm in the average reprojection error based on two separate evaluation methods. We further perform experiments with the internal tracking on the Epson Moverio BT-300 to demonstrate that the method can provide similar results with other HMDs.

Alignment of the Virtual Scene to the Tracking Space of a Mixed Reality Head-Mounted Display

TL;DR

This paper proposes a calibration procedure to properly align the coordinate system of a 3D virtual scene that the user sees with that of the tracker, and performs experiments with Microsoft HoloLens, taking advantage of its self localization and spatial mapping capabilities to eliminate the requirement for line of sight from the HMD to the object or external tracker.

Abstract

With the mounting global interest for optical see-through head-mounted displays (OST-HMDs) across medical, industrial and entertainment settings, many systems with different capabilities are rapidly entering the market. Despite such variety, they all require display calibration to create a proper mixed reality environment. With the aid of tracking systems, it is possible to register rendered graphics with tracked objects in the real world. We propose a calibration procedure to properly align the coordinate system of a 3D virtual scene that the user sees with that of the tracker. Our method takes a blackbox approach towards the HMD calibration, where the tracker's data is its input and the 3D coordinates of a virtual object in the observer's eye is the output; the objective is thus to find the 3D projection that aligns the virtual content with its real counterpart. In addition, a faster and more intuitive version of this calibration is introduced in which the user simultaneously aligns multiple points of a single virtual 3D object with its real counterpart; this reduces the number of required repetitions in the alignment from 20 to only 4, which leads to a much easier calibration task for the user. In this paper, both internal (HMD camera) and external tracking systems are studied. We perform experiments with Microsoft HoloLens, taking advantage of its self localization and spatial mapping capabilities to eliminate the requirement for line of sight from the HMD to the object or external tracker. The experimental results indicate an accuracy of up to 4 mm in the average reprojection error based on two separate evaluation methods. We further perform experiments with the internal tracking on the Epson Moverio BT-300 to demonstrate that the method can provide similar results with other HMDs.

Paper Structure

This paper contains 22 sections, 8 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. OST-HMD Calibration
  4. Tracking Systems
  5. Method
  6. Calibration with blackbox approach
  7. Multipoint Alignment Using a Single 3D object
  8. Implementation
  9. Calibration with Head-Anchored Tracking System
  10. Calibration with World-Anchored Tracking System
  11. Experiments
  12. Standard Method (One Corner Alignment)
  13. Multipoint Single 3D Object Method
  14. Evaluation
  15. Results
  16. ...and 7 more sections

Figures (12)

  • Figure 1: Conceptual schematic of blackbox approach towards display calibration: regardless of the internal features of an HMD, as long as there is access to the generated 3D scene for the user and the tracking data, the display can be calibrated with the proposed method.
  • Figure 2: Left: tracked object; Right: Multipoint single object alignment procedure.
  • Figure 3: Implementation of the calibration with head-anchored tracking system. The HoloLens embedded RGB camera is used as the head-anchored tracker. The transformation map of the calibration procedure is demonstrated.
  • Figure 4: Implementation of the calibration with world-anchored tracking system (fusionTrack 500). Passive spherical markers form a frame that is attached to the colored cube for tracking.
  • Figure 5: The workflow of calibration with head-anchored tracker and world-anchored tracker
  • ...and 7 more figures