Fast-Forward Reality: Authoring Error-Free Context-Aware Policies with Real-Time Unit Tests in Extended Reality

TL;DR

This work tackles the reliability gap in end-user context-aware policy (CAP) authoring by introducing Fast-Forward Reality, an Extended Reality (XR) workflow that enables in-situ authoring, automatic generation of personalized unit-test-like test cases from a user’s context history, and immersive validation. The system implements an author-test-refine loop, guided by a test-case generation algorithm that leverages the uncertainty coefficient $theils\_u$ to diversify scenarios while keeping tests concise. An XR interface provides immersive, in-situ visualizations to help users interpret test cases and refine CAPs accordingly. In a user study with N=12, CAPs authored with the framework achieved higher precision (90.6%), recall (83.3%), and overall F-score (85.4%) than a baseline, with strong usability feedback (SUS ~86) and positive qualitative remarks on immersion and trust. The results suggest that coupling personalized test-case generation with XR-based validation can make end-user CAP authoring more accurate, intuitive, and scalable for real-world smart environments, while highlighting future work on sensing robustness, multi-CAP conflicts, and richer interaction modalities.

Abstract

Advances in ubiquitous computing have enabled end-user authoring of context-aware policies (CAPs) that control smart devices based on specific contexts of the user and environment. However, authoring CAPs accurately and avoiding run-time errors is challenging for end-users as it is difficult to foresee CAP behaviors under complex real-world conditions. We propose Fast-Forward Reality, an Extended Reality (XR) based authoring workflow that enables end-users to iteratively author and refine CAPs by validating their behaviors via simulated unit test cases. We develop a computational approach to automatically generate test cases based on the authored CAP and the user's context history. Our system delivers each test case with immersive visualizations in XR, facilitating users to verify the CAP behavior and identify necessary refinements. We evaluated Fast-Forward Reality in a user study (N=12). Our authoring and validation process improved the accuracy of CAPs and the users provided positive feedback on the system usability.

Figures (6)

• Figure 1: The authoring workflow of Fast-Forward Reality. While being immersed in the authoring environment, a user starts authoring to define a target action and initial context instances. The system generates test cases from the user-authored CAP and context history. When the user starts unit testing, test cases are visualized to enable validation of the CAP's behavior. If unexpected instances are identified, the user refines the original CAP by editing the context instances. The user repeats this process to iteratively refine the CAP until it meets their expectations.
• Figure 2: The framework of authoring CAP using Fast-Forward Reality. (a) A user's everyday contexts are recorded as the context history, which contains a series of context scenes where each one is a collection of concurrently occurring context instances of corresponding context factors that can be detected in the environment. (b) In a user-authored CAP, the 'trigger' contains multiple context instances from the same or different context factors, while the 'action' is a context instance of a smart object that reflects the functional state (e.g., 'TV is on'). (c) During deployment, for all the context factors included in the CAP, when the specified context instances are present in the real-time context scene, the 'action' is triggered.
• Figure 3: Test case generation algorithm. For each context factor, the algorithm starts with an empty test case. In the correlation assessment stage, we investigate whether the processing context factor holds a high uncertainty coefficient value with the target action, and whether it is already selected into the CAP. Depending on the conditions, the algorithm then processes the concurrency assessment stage to select the context instances into the test case. In one specific condition, the system will skip this process and directly iterate to the next context factor, while in other conditions, the algorithm outputs a test case and starts to process the next context factor.
• Figure 4: The XR-based authoring interface of Fast-Forward Reality. (a) The main menu rendered on a user's non-dominant hand enables users to 'edit' existing CAPs, 'add' a new CAP and author it, 'start validation' of unit test cases, 'delete', and 'save' the CAP. (b) An authoring panel displayed on the user's non-dominant hand indicates the action and triggers in the current CAP. (c-1) The immersive XR authoring environment. (c-2) Location and activity context instances are represented using avatars with different poses. (c-3) Spatial states are placed in situ, next to corresponding smart objects. (c-4) The digital state and user state are rendered on the user's hand. All context instances are color-coded to represent different conditions where blue color indicates that the context instance is not selected, pink represents that it is included in the current CAP, and red represents that it is included in the current test case.
• Figure 5: (a) The virtual environment used for the user study. (b) The context history collection tool. The top left corner below the 'new/save sequence' button contains spatial-insensitive contexts, time and user-specific digital/user states (e.g., 'is working', 'feel tired', 'consume too many coffees', and 'not enough water in the coffee machine') (c-1) The floor plan of the target environment. By clicking each blue dot, pre-designated spatial-sensitive context instances such as object states and activities can be toggled together with their locations (e.g., 'coffee machine is on/off', 'stay', 'cooking', and 'eating'.) (c-2) The available context instances that are visualized in different icons according to the context factors. Typically, location is spatially overlaid on the floor plan, while available activities and object states are displayed as lists after selecting the location. (c-3) The timeline indicating all the recorded context instances. Explanation of the context instance will appear when hovering on the icon. One can delete a context instance by clicking the block/dot.