MineXR: Mining Personalized Extended Reality Interfaces

TL;DR

MineXR addresses the challenge of understanding what XR content users want in everyday contexts by enabling in-situ, bottom-up elicitation of personalized XR layouts. It couples a smartphone-based widget creation workflow with AR placement and a cloud-backed HMD preview to collect rich interaction data, culminating in a dataset of 109 XR layouts and 695 widgets from 31 participants across four environments. The work introduces an integrated data-analysis toolchain (scene reconstruction and annotation) and derives design guidelines toward function-based, context-aware XR interfaces, with potential for adaptive and computational XR applications. By open-sourcing both the data and tools, MineXR provides a foundation for advancing personalized XR UI design and evaluation in real-world settings.

Abstract

Extended Reality (XR) interfaces offer engaging user experiences, but their effective design requires a nuanced understanding of user behavior and preferences. This knowledge is challenging to obtain without the widespread adoption of XR devices. We introduce MineXR, a design mining workflow and data analysis platform for collecting and analyzing personalized XR user interaction and experience data. MineXR enables elicitation of personalized interfaces from participants of a data collection: for any particular context, participants create interface elements using application screenshots from their own smartphone, place them in the environment, and simultaneously preview the resulting XR layout on a headset. Using MineXR, we contribute a dataset of personalized XR interfaces collected from 31 participants, consisting of 695 XR widgets created from 178 unique applications. We provide insights for XR widget functionalities, categories, clusters, UI element types, and placement. Our open-source tools and data support researchers and designers in developing future XR interfaces.

Figures (17)

• Figure 1: Architecture of the MineXR data collection and analysis platform.
• Figure 2: Widget creation and placement workflow. MineXR provides a smartphone-based widget creation and XR placement workflow. A participant can create new widgets from screenshots of mobile apps or websites. Then, they can create XR widget layouts by moving the phone to the desired widget location and tapping on the screen to place.
• Figure 3: Example screenshots taken from MineXR's interactive scene reconstruction Unity plugin, including a participant-generated layout. Researchers can reconstruct the spatial layout of each scenario with an optional overlay of the environment's 3D scan or semantic map. The 3D scan (a,b) in this example was generated using Polycam, the semantic map (c) using Apple RoomPlan.
• Figure 4: Web-based data annotation and summary interface. The annotation interface (left) loads each instance data of widget placement and displays the corresponding anchor ID, participant ID, environment, task, screenshot image, and widget image of the widget. The interface provides a form interface including fields for data annotation, e. g., application name, screenshot description, widget description, excluded parts, type of UI element(s), application category, etc. Below the purple annotation box is an interactive data table where researchers can browse through the annotations by search, filters, and sorts. The summary interface (right) provides basic statistical summary and visualizations of the annotated data.
• Figure 5: The four environments in which we collected data. Each participant created XR layouts in two of the four environments.