InteRecon: Towards Reconstructing Interactivity of Personal Memorable Items in Mixed Reality

TL;DR

This paper introduces Interactive Digital Item (IDI), a framework for digitizing personal mementos while preserving their real-world interactivity. It reports a formative study identifying geometry, interfaces, and embedded content as core attributes, and presents InteRecon, an AR prototype that reconstructs 3D appearance, physical transforms, interfaces, and embedded content. A two-session user study with 16 participants demonstrates the feasibility, expressiveness, and user acceptance of IDI creation, revealing that high-fidelity reconstructions enhance ownership and memory realism, and that IDI has potential to enrich memory archives and support new use cases. The work discusses practical implications, humane considerations, and future directions for broader technical development and ecosystem growth around user-generated, interactive digital artifacts in mixed reality.

Abstract

Digital capturing of memorable personal items is a key way to archive personal memories. Although current digitization methods (e.g., photos, videos, 3D scanning) can replicate the physical appearance of an item, they often cannot preserve its real-world interactivity. We present Interactive Digital Item (IDI), a concept of reconstructing both the physical appearance and, more importantly, the interactivity of an item. We first conducted a formative study to understand users' expectations of IDI, identifying key physical interactivity features, including geometry, interfaces, and embedded content of items. Informed by these findings, we developed InteRecon, an AR prototype enabling personal reconstruction functions for IDI creation. An exploratory study was conducted to assess the feasibility of using InteRecon and explore the potential of IDI to enrich personal memory archives. Results show that InteRecon is feasible for IDI creation, and the concept of IDI brings new opportunities for augmenting personal memory archives.

Figures (7)

• Figure 1: The interactive process for reconstructing 3D appearance. (1) The user opens the application and points it at the target object. (2) An automatic bounding box is generated around the target object in the application interface. (3) Circle around the object with visual guidance on regions to capture more images. (4) A 3D reconstructed model is ready for further use.
• Figure 2: The interactive process of adding physical transforms. (a-1,2) Segment the puppy's model using an automatic approach. (b-1,2) Segment the puppy model in AR by using a segmenting plane and breaking down the model's leg along the plane. (c) Use hands to touch the blue cube to observe the movement features of each demonstrated joint until identify one that resembles the model's leg. Detailed introductions to each joint are listed in Table \ref{['tab:joints']}, and the numbers close to each joint in the figure correspond to the numbers in Table \ref{['tab:joints']}. (d) Select the 'movable' segment of the model (the leg) using a pinching gesture and confirm the choice by clicking the 'movable' button, which is analogous when selecting the 'base' (the body). Press the 'Apply' button to apply the joint to the model's leg and body. (e) After repeating the above processes for mapping the model's ear joints, shake the model's body and generate similar motions to a real-world toy puppy's legs and ears in AR.
• Figure 3: The AR interface for the functions of reconstructing the interface and adding embedded content. (a) The 'interface' in the menu includes four commonly used widget categories: 'knob', 'screen', 'slider', and 'button', and a target model category that the user scanned by Function 1 (in this case, the TV model). Press the 'knob' and 'screen' buttons to select a knob widget and a screen widget and attach them to the TV model to reconstruct its interface. (b) The 'content' includes three categories in the menu: 'video', 'audio', and 'picture'. Press the 'video' button to release the three embedded videos that users uploaded and drag them to the TV model to import.
• Figure 4: The mean duration time of each atomic interaction in three tasks in Session One.
• Figure 5: Average subjective rating scores for 4 categories of functions in Table \ref{['tab:atomic_interaction']}. The first (green), second (orange), third (purple), and fourth (pink) columns in each cluster indicate the score distribution across four function categories. 1 - strongly disagree, 7 - strongly agree.