Geometry Aware Passthrough Mitigates Cybersickness

TL;DR

This work tackles cybersickness in Video See-Through headsets by proposing Geometry Aware Passthrough (GAP), a depth-based reprojection approach that better aligns camera imagery with the user's eye view. It introduces benchmarking metrics for geometric accuracy and warping, plus a protocol to holistically assess visually-induced discomfort in real-world VST tasks. A within-subject user study (n=25) shows GAP significantly reduces nausea, disorientation, and total cybersickness compared with direct passthrough, while preserving comparable task performance and improving spatial awareness, albeit with increased warping artifacts. The findings inform VST HMD design, highlight trade-offs between geometric fidelity and warping, and provide a practical framework for evaluating passthrough comfort and guiding future XR improvements.

Abstract

Virtual Reality headsets isolate users from the real-world by restricting their perception to the virtual-world. Video See-Through (VST) headsets address this by utilizing world-facing cameras to create Augmented Reality experiences. However, directly displaying camera feeds causes visual discomfort and cybersickness due to the inaccurate perception of scale and exaggerated motion parallax. This paper demonstrates the potential of geometry aware passthrough systems in mitigating cybersickness through accurate depth perception. We first present a methodology to benchmark and compare passthrough algorithms. Furthermore, we design a protocol to quantitatively measure cybersickness experienced by users in VST headsets. Using this protocol, we conduct a user study to compare direct passthrough and geometry aware passthrough systems. To the best of our knowledge, our study is the first one to reveal significantly reduced nausea, disorientation, and total scores of cybersickness with geometry aware passthrough. It also uncovers several potential avenues to further mitigate visually-induced discomfort.

Figures (9)

• Figure 1: Comparison between reprojection in direct and geometry aware passthrough. In this figure, we show a 2D illustration to describe the reprojection step where foreground and background are reprojected using different geometries. For direct passthrough, each point in 3D is assumed to belong to a single plane situated at a certain distance. However, that results in reprojecting objects at incorrect pixel locations and at incorrect scale (left). In comparison, geometry aware passthrough uses a geometry estimate which improves the perceived location and the scale (right).
• Figure 2: DP versus GAP output images. Two images are shown above taken from the headset placed at the same point in the scene. We observe that DP enlarges all the objects, making the scene look closer to the user. The scale difference can be easily noticed when comparing the text on the bottom right of both images. While GAP improves the scale, it also results in warping artifacts which can be observed on the edge of the door behind the hand.
• Figure 3: Spatial Reprojection Error. Evaluation of perceived scene geometry with reprojection error. We take the Estimated Depth and GT Depth for the same image, and reproject every pixel to the eye view. The differences in the reprojected locations is used as the evaluation criteria. On the right, we compare the effects of smoothing the depth. Since the estimated depth can have errors, warping and rotating this depth map to the right camera can further exacerbate these errors. Gaussian smoothing of the estimated depth helps reduce these errors on the right eye as shown above.
• Figure 4: Geometrical Errors. Comparison of the errors in depth estimation for different GAP variants and DP. We observe that smoothing increases the depth errors. DP on the other hand has the highest error among all approaches achieving minimum depth error at the 2m (2000mm) mark which is expected since we assume all the points are at 2m for DP.
• Figure 5: Warping Errors. We consider planar targets with known hand-crafted textures to quantify warping artifacts in the passthrough images. Specifically, we take the passthrough reprojected image and obtain the crop around the reprojected texture of interest using ArUco aruco marker detectors (see left tile). Then we obtain the correspondences between the known texture and the reprojected texture using ML-based keypoint matching techniques sarlin20superglue (a). We then use a homography solver to find the residual errors which measures the deviation of the reprojected image from a planar surface. This residual error can be plotted on the image (b) indicating pixel locations where the warping is observed. We also observe that geometry aware passthrough has a higher warping error than direct passthrough (c).