3D-Mirrorcle: Bridging the Virtual and Real through Depth Alignment in AR Mirror Systems

TL;DR

The paper addresses depth misalignment in AR mirrors caused by parallax between the 3D mirror reflection and the 2D screen display. It introduces a hardware-software co-design, combining a lenticular grating, real-time Mirror Reflection Alignment, and Lenticular Grating Segmentation to render depth-matched, glasses-free AR on a mirror surface. A Leia Lume Pad 2 prototype and a 36-participant user study demonstrate significant improvements in accuracy ($24.72\%$), immersion ($31.4\%$), and user satisfaction ($44.4\%$) over 2D smart mirrors. The approach offers practical depth-aware AR for applications in makeup, fashion, education, and interactive experiences within everyday mirror use.

Abstract

Smart mirrors have emerged as a new form of augmented reality (AR) interface for home environments. However, due to the parallax in human vision, one major challenge hindering their development is the depth misalignment between the 3D mirror reflection and the 2D screen display. This misalignment causes the display content to appear as if it is floating above the mirror, thereby disrupting the seamless integration of the two components and impacting the overall quality and functionality of the mirror. In this study, we introduce 3D-Mirrorcle, an innovative augmented reality (AR) mirror system that effectively addresses the issue of depth disparity through a hardware-software co-design on a lenticular grating setup. With our implemented real-time position adjustment and depth adaptation algorithms, the screen display can be dynamically aligned to the user's depth perception for a highly realistic and engaging experience. Our method has been validated through a prototype and hands-on user experiments that engaged 36 participants, and the results show significant improvements in terms of accuracy (24.72% $\uparrow$), immersion(31.4% $\uparrow$), and user satisfaction (44.4% $\uparrow$) compared to the existing works.

Figures (10)

• Figure 1: (a) Parallax in human vision. (b) Depth mismatch in 2D smart mirrors: the mirror reflection blurs when viewing the screen display, and the screen blurs vice versa. (c) Two separate images are displayed for each eye to achieve depth perception.
• Figure 2: The Pipeline of 3D-Mirrorcle: we locate the coordinates of key points in reality using a depth camera and then process the captured image to achieve an AR effect. After creating a 3D display through lenticular grating and segmented images, viewers can clearly see both their own reflections and the AR effects on the display.
• Figure 3: Due to parallax, each eye sees point P' from a distinct angle. To align point Q with P', we create parallax on the screen by showing points $Q_\textit{left}$ and $Q_\textit{right}$ at the sight-screen intersection, each visible to only one eye via lenticular grating, merging to create the illusion of alignment.
• Figure 4: The display is divided into interlaced strips. Through the refraction of a lenticular grating, the left eye can only see the blue strips, while the right eye can only see the orange strips.
• Figure 5: User Study Setups: (a) two mirrors surround the user to compare. (b) left (2D smart mirror): the user can see a guidance image on the screen through a half-mirror. (c) right (3D-Mirrorcle): user's each eye sees one of the images and merges into one depth-aligned image on the face.