Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion

TL;DR

This work tackles the challenge of capturing and reconstructing human–scene interactions using only wearable sensors. By combining egocentric head-camera localization with IMU-based pose estimation, a Transformer-based contact predictor, and a contact-driven object-motion model, iReplica produces a coherent virtual replica of both human motion and scene changes. The authors introduce two new datasets (H-contact and EgoHOI), demonstrate strong gains in contact detection accuracy and object localization, and show that contact-aware trajectory correction yields more plausible interactions than strong baselines. The approach lays groundwork for ego-centric AR/VR and digital twins in dynamic environments, while acknowledging limitations such as lack of physics and dependence on a predefined interactive environment. Overall, iReplica advances the feasibility of end-to-end wearable-sensor–driven modeling of dynamic scenes and human actions.

Abstract

Our world is not static and humans naturally cause changes in their environments through interactions, e.g., opening doors or moving furniture. Modeling changes caused by humans is essential for building digital twins, e.g., in the context of shared physical-virtual spaces (metaverses) and robotics. In order for widespread adoption of such emerging applications, the sensor setup used to capture the interactions needs to be inexpensive and easy-to-use for non-expert users. I.e., interactions should be captured and modeled by simple ego-centric sensors such as a combination of cameras and IMU sensors, not relying on any external cameras or object trackers. Yet, to the best of our knowledge, no work tackling the challenging problem of modeling human-scene interactions via such an ego-centric sensor setup exists. This paper closes this gap in the literature by developing a novel approach that combines visual localization of humans in the scene with contact-based reasoning about human-scene interactions from IMU data. Interestingly, we can show that even without visual observations of the interactions, human-scene contacts and interactions can be realistically predicted from human pose sequences. Our method, iReplica (Interaction Replica), is an essential first step towards the egocentric capture of human interactions and modeling of dynamic scenes, which is required for future AR/VR applications in immersive virtual universes and for training machines to behave like humans. Our code, data and model are available on our project page at http://virtualhumans.mpi-inf.mpg.de/ireplica/

Figures (11)

• Figure 1: Problem subdivision. We demonstrate that joint integration of different sub-research problem improves and support each other. We show this is fundamental to achieve our goal of estimating human-scene interaction from wearable sensors only.
• Figure 2: Challenges. Top row: The prediction of human-scene contacts (red circles) is hard because the interactions are frequently not in the camera view. Bottom row: Virtual replica of human pose and localization by prior work HPS guzov2021human. HPS achieved great progress in localizing humans solely by wearable sensors (camera+IMUs). However, for our task, the localization error of 4--16 cm (red lines) leads to visually implausible results for scene interactions.
• Figure 3: Overview of iReplica. iReplica estimates a subject's location and full pose within a large 3D scene and dynamically track changes made to the scene by the subject -- using only wearable sensors. We do so in 4 steps: A) We obtain an initial localization of the subject in the IE by head camera self-localization. B) The start of the interaction is predicted by a neural network. Predictions are provided as contact / no-contact classification of the subject's hands (red and blue areas). The contacts are used to correct head camera localization of the subject, snapping the human trajectory smoothly to the object. C) The motion of the contacted regions is used to infer the object trajectory (green). D) The network predicts the release, essential to stop object dragging. The algorithm is detailed in Sec. \ref{['sec:ireplica']}.
• Figure 4: Contact prediction based on human pose. Interactions are frequently unobserved in an egocentric view, see Fig. \ref{['fig:challenges']} (top row), making contact prediction ill-posed. Instead, we propose to predict from sequences of full 3D human poses. We leverage a transformer-based architecture that takes 61 frames {$i-30, \dots , i+30$} of SMPL pose vectors of size $S=69$ and predicts the contact probability for each hand for the middle frame $i$. See Sec. \ref{['sec:contact']} for details.
• Figure 5: Qualitative results. We show three examples of human interaction, pairing the head-mounted camera view with the interaction modeling achieved by iReplica. The object is not always visible during the interaction (Interaction 1), hand grasping can be difficult to understand from the camera (Interaction 2), or object occludes a majority of the first person view (Interaction 3). By relying on human-centric contact detection, iReplica achieves reliable modeling in all these challenging scenarios. Please see our video for more results.