No Fuss, Just Function -- A Proposal for Non-Intrusive Full Body Tracking in XR for Meaningful Spatial Interactions

TL;DR

The paper addresses the XR interaction gap where natural input often relies on handheld controllers and proposes non-intrusive full-body tracking (FBT) using markerless Human Pose Estimation (HPE) as an accessible alternative. It outlines design conditions (no wearables; inconspicuous tracking) and discusses headset-integrated or RGB/depth-based sensing to enable inclusive interactions, including adaptations for tremors and motor limitations. Although conceptual, the work surveys related literature and frames an evaluation-driven research agenda, emphasizing usability, workload, and accessibility as core metrics. The proposed approach has potential to broaden MR accessibility, simplify setup, and foster natural 3D spatial interactions for a wider range of users.

Abstract

Extended Reality (XR) is a rapidly growing field with a wide range of hardware from head mounted displays to installations. Users have the possibility to access the entire Mixed Reality (MR) continuum. Goal of the human-computer-interaction (HCI) community is to allow natural and intuitive interactions but in general interactions for XR often rely on handheld controllers. One natural interaction method is full body tracking (FBT), where a user can use their body to interact with the experience. Classically, FBT systems require markers or trackers on the users to capture motion. Recently, there have been approaches based on Human Pose Estimation (HPE), which highlight the potential of low-cost non-intrusive FBT for XR. Due to the lack of handheld devices, HPE may also improve accessibility with people struggling with traditional input methods. This paper proposes the concept of non-intrusive FBT for XR for all. The goal is to spark a discussion on advantages for users by using a non-intrusive FBT system for accessibility and user experience.