Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hologram: Realtime Holographic Overlays via LiDAR Augmented Reconstruction

Ekansh Agrawal

TL;DR

The paper tackles real-time holographic overlays on consumer hardware by proposing LiDAR-augmented reconstruction for metric facial 3D reconstructions on mobile devices. It evaluates three approaches: monocular depth estimation, LiDAR+TrueDepth fusion with upscaling, and template-modeling via MICA, highlighting the trade-offs between fidelity, speed, and deployability. Key findings show that LiDAR+TrueDepth fusion can reach practical frame rates with high fidelity but struggles with occlusions and single-device constraints, while monocular methods suffer from non-metric depth and low FPS, and template modeling faces deployment challenges. The work demonstrates the potential for mobile holography to enable AR/telepresence and entertainment, while outlining concrete avenues for hardware integration and further software maturation to realize practical, portable real-time holographic overlays.

Abstract

Guided by the hologram technology of the infamous Star Wars franchise, I present an application that creates real-time holographic overlays using LiDAR augmented 3D reconstruction. Prior attempts involve SLAM or NeRFs which either require highly calibrated scenes, incur steep computation costs, or fail to render dynamic scenes. I propose 3 high-fidelity reconstruction tools that can run on a portable device, such as a iPhone 14 Pro, which can allow for metric accurate facial reconstructions. My systems enable interactive and immersive holographic experiences that can be used for a wide range of applications, including augmented reality, telepresence, and entertainment.

Hologram: Realtime Holographic Overlays via LiDAR Augmented Reconstruction

TL;DR

The paper tackles real-time holographic overlays on consumer hardware by proposing LiDAR-augmented reconstruction for metric facial 3D reconstructions on mobile devices. It evaluates three approaches: monocular depth estimation, LiDAR+TrueDepth fusion with upscaling, and template-modeling via MICA, highlighting the trade-offs between fidelity, speed, and deployability. Key findings show that LiDAR+TrueDepth fusion can reach practical frame rates with high fidelity but struggles with occlusions and single-device constraints, while monocular methods suffer from non-metric depth and low FPS, and template modeling faces deployment challenges. The work demonstrates the potential for mobile holography to enable AR/telepresence and entertainment, while outlining concrete avenues for hardware integration and further software maturation to realize practical, portable real-time holographic overlays.

Abstract

Guided by the hologram technology of the infamous Star Wars franchise, I present an application that creates real-time holographic overlays using LiDAR augmented 3D reconstruction. Prior attempts involve SLAM or NeRFs which either require highly calibrated scenes, incur steep computation costs, or fail to render dynamic scenes. I propose 3 high-fidelity reconstruction tools that can run on a portable device, such as a iPhone 14 Pro, which can allow for metric accurate facial reconstructions. My systems enable interactive and immersive holographic experiences that can be used for a wide range of applications, including augmented reality, telepresence, and entertainment.
Paper Structure (14 sections, 2 equations, 11 figures)

This paper contains 14 sections, 2 equations, 11 figures.

Table of Contents

  1. Motivation
  2. Prior Works
  3. Simultaneous Localization and Mapping
  4. Depth Estimation
  5. Neural Radiance Fields
  6. Methodology
  7. Attempt 1: Monocular Depth Estimation
  8. Attempt 2: LiDAR + TrueDepth
  9. Attempt 3: Template Modeling
  10. Results
  11. Attempt 1: Monocular Depth Estimation
  12. Attempt 2: LiDAR + TrueDepth
  13. Attempt 3: Template Modeling
  14. Further Study

Figures (11)

  • Figure 1: Hologram technology from the Star Wars universe
  • Figure 2: Attempt 1 utilizes monocular depth estimation alongside classical image projections in order to transfer pixels from their image representation into there voxel representation.
  • Figure 3: Attempt 2 utilizes the LiDAR and TrueDepth data streamed from the iPhone 14 Pro's front facing sensors. This data is then upscaled with a SRCNN model trained on facial depth maps. I fuse this data and then use classical projections using the phone's IMU sensor for camera poses.
  • Figure 4: Attempt 3
  • Figure 5: Projected light coordinates for the TrueDepth and LiDAR sensors shipped with the Apple iPhones.
  • ...and 6 more figures