SimulataR: Rapid Assisted Reality Prototyping using Design-Blended Videos

TL;DR

This work tackles the challenge of rapidly prototyping Assisted Reality (aR) user interfaces for OST-HMDs across varied real-world contexts. It introduces SimulataR, a desktop-based approach that creates design-blended FPV videos by overlaying UI designs onto real context videos with corrections for light and field of view, enabling realistic previews without expensive hardware. An empirical field study with 12 AR users shows SimulataR can closely approximate OST-HMD viewing in indoor and low-to-moderate outdoor lighting, though performance degrades under bright outdoor conditions, especially for Hololens 2. Case studies with two designers demonstrate the tool’s value for early-stage iterative design, offering a practical pipeline for rapid prototyping and context-aware evaluation that can reduce real-world testing effort and social friction.

Abstract

Assisted Reality (aR) is a subfield of Augmented Reality (AR) that overlays information onto a user's immediate view via see-through head-mounted displays (OST-HMDs). This technology has proven to be effective and energy-efficient to support the user and information interaction for everyday wearable intelligent systems. The aR viewing experience, however, is affected by varying real-world backgrounds, lighting, and user movements, which makes designing for aR challenging. Designers have to test their designs in-situ across multiple real-world settings, which can be time-consuming and labor-intensive. We propose SimulataR, a cost-effective desktop-based approach for rapid aR prototyping using first-person-view context videos blended with design prototypes to simulate an aR experience. A field study involving 12 AR users comparing SimulataR to real OST-HMDs found that SimulataR can approximate the aR experience, particularly for indoors and in low-to-moderate lit outdoor environments. Case studies with two designers who used SimulataR in their design process demonstrates the potential of design-blended videos for rapid aR prototyping.

Figures (4)

• Figure 1: (a) Comparison of a simple overlay of content on a context video (left half) and a design-blended video with visual corrections (right half) and (b) An overview of the routes undertaken by the user
• Figure 2: Descriptive table of equivalence in users’ assessments for the design dimensions in each context. Each cell is marked green if there is significant equivalence of ratings for both design variants, yellow if equivalence was seen only for one of the variants, and red if equivalence failed for both variants.
• Figure 3: Design-blending tool used for the case study
• Figure 4: The design process adopted by designer D1 in the case study. They began with testing on a Hololens 2, followed by testing on SimulataR to make adjustments in the color, size, and layout of the UIs. The designs were subsequently verified in diverse contexts.