Tell-XR: Conversational End-User Development of XR Automations

TL;DR

Tell-XR tackles the challenge of enabling non-programmers to author XR automations by using an LLM-driven agent that guides users through event-condition-action rules. A formative Wizard-of-Oz study identifies a Define-Explore-Refine-Confirm-Export dialogue flow and a three-component architecture (UI, Automation Engine on Home Assistant, and Tell-XR Bot) with multimodal inputs. An evaluation in a VR museum and an AR smart home demonstrates generally high task success, low workload, and positive user experience, while revealing hallucination risks and modification challenges that require improvement. The work suggests a generalizable approach for end-user XR authoring across VR/AR domains and IoT-enabled environments, with implications for scalable, explainable XR customization.

Abstract

The availability of extended reality (XR) devices has widened their adoption, yet authoring interactive experiences remains complex for non-programmers. We introduce Tell-XR, an intelligent agent leveraging large language models (LLMs) to guide end-users in defining the interaction in XR settings using automations described as Event-Condition-Action (ECA) rules. Through a formative study, we identified the key conversation stages to define and refine automations, which informed the design of the system architecture. The evaluation study in two scenarios (a VR museum and an AR smart home) demonstrates the effectiveness of Tell-XR across different XR interaction settings.

Figures (4)

• Figure 1: Transitions among the identified stages (Define, Explore, Refine, Confirm and Export) in the collected conversations during the formative study in the VR museum (left) and AR home (right). Rows are the starting stages, columns the targets. Values are normalised on the total number of conversation turns.
• Figure 2: Examples of the Tell-XR interface. The VR museum environment (A), the sceptre in the socket condition (B) and confirming the example automation (C). In the AR home environment, highlighting the interactable devices (D), showing the light capabilities (E) and confirming the modified automation (F).
• Figure 3: The Tell-XR architecture consists of three components, 1) the User Interface (light blue), which provides the multimodal input, 2) the Automation Engine, currently implemented on top of Home Assistant home-assistant and 3) the Tell-XR bot, consisting of several sub-modules to handle the phases in the conversation identified in Section \ref{['sec:formative-findings']}
• Figure 4: Summary of the quantitative data collected during the user test through (from top to bottom) NASA-TLX hart1988, BUS-11 bus11, UEQ-short ueq-s questionnaires and task completion time. For each dimension we report the VR values on the left (light-blue in box-plots) and the AR values on the right (orange).