Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands

TL;DR

The results show subjects are less sensitive to latency artifacts and perform faster and with more ease a given associated task over the naïve approach of directly projecting rendered frames from the 3D pose estimation.

Abstract

We present a technique for dynamically projecting 3D content onto human hands with short perceived motion-to-photon latency. Computing the pose and shape of human hands accurately and quickly is a challenging task due to their articulated and deformable nature. We combine a slower 3D coarse estimation of the hand pose with high speed 2D correction steps which improve the alignment of the projection to the hands, increase the projected surface area, and reduce perceived latency. Since our approach leverages a full 3D reconstruction of the hands, any arbitrary texture or reasonably performant effect can be applied, which was not possible before. We conducted two user studies to assess the benefits of using our method. The results show subjects are less sensitive to latency artifacts and perform faster and with more ease a given associated task over the naive approach of directly projecting rendered frames from the 3D pose estimation. We demonstrate several novel use cases and applications.

Figures (11)

• Figure 1: Pipeline overview. Top: for every projected frame (at time $T$) our main thread generates and skins a 3D mesh using an extrapolated 3D pose estimation (\ref{['sec:motion_to_photon']}), deforms the rendered mesh in 2D to reduce static bias using 2D landmark motion estimation (\ref{['sec:mls_deformation']}), and performs a novel perceptual boundary reduction step (\ref{['sec:pbr']}) which decreases the perceived latency, extending the projected surface area to match precise hand position rapidly. The result is projected onto the hand. Bottom: three asynchronous threads are responsible for filling the buffers with information, each operating asynchronously at different rates.
• Figure 2: Physical setup. The setup consists of a coaxial high speed projector and camera, diffuse IR illumination, a hot mirror, and a Leap Motion Controller, held together on a configurable rig.
• Figure 3: MLS deformation. Left: due to accumulated errors, the rendered hand generated from the 3D pose and the camera frame exhibit alignment errors. Right: we reduce the bias by applying a deformation grid in screen space, computed from a set of source points $Q$ (LMC Joints) and their target $P$ (2D landmarks).
• Figure 4: Illustration of the PBR step while the hand is translating to the right. Top: given the up-to-date camera frame, a rendered hand and their intersection, we compute for each pixel $p_C \in \mathcolor{regc}{\Omega_C}$ the nearest neighbour $p_D \in \mathcolor{regd}{\Omega_D}$, and find a reflection about the neighbour $p'_D$. We then compute the UV coordinates of $p_C$ by extrapolating the UV coordinates of $p'_D$ and $p_D$. Bottom: comparing simulated frames when using versus not using the PBR step.
• Figure 5: PBR During Rapid Motion. Left: the hand is translating from left to right. Right: the hand is deforming into the "OK" gesture. When using PBR, a visually plausible higher coverage of the hand is achieved both for static and moving hands. Note: the projected image in all cases is first deformed using the MLS step for better alignment.