An Interaction Design Toolkit for Physical Task Guidance with Artificial Intelligence and Mixed Reality

TL;DR

This work addresses the design challenges of AI-enabled Mixed Reality task guidance by introducing MixITS-Kit, a targeted design toolkit built from eight low-fidelity MixITS prototypes developed in a 10-week graduate course. The toolkit combines an Interaction Canvas, six design considerations, and 36 design patterns to help designers analyze gulfs of execution and evaluation across user, AI, and environment, supporting multi-level reasoning from high-level goals to concrete solutions. An asynchronous, multi-task evaluation with eight participants demonstrates the toolkit’s potential to provide a shared vocabulary and actionable guidance, though some users found abstraction level and pattern mapping challenging. The study argues that MixITS-Kit can accelerate the development of safe, context-aware, and learnable AI-assisted MR systems, with implications for education, practice, and future tool refinement. Overall, the Toolkit promises to streamline the design of embodied, real-environment AI guidance, enabling broader accessibility to skill acquisition and safety-critical tasks.

Abstract

Physical skill acquisition, from sports techniques to surgical procedures, requires instruction and feedback. In the absence of a human expert, Physical Task Guidance (PTG) systems can offer a promising alternative. These systems integrate Artificial Intelligence (AI) and Mixed Reality (MR) to provide realtime feedback and guidance as users practice and learn skills using physical tools and objects. However, designing PTG systems presents challenges beyond engineering complexities. The intricate interplay between users, AI, MR interfaces, and the physical environment creates unique interaction design hurdles. To address these challenges, we present an interaction design toolkit derived from our analysis of PTG prototypes developed by eight student teams during a 10-week-long graduate course. The toolkit comprises Design Considerations, Design Patterns, and an Interaction Canvas. Our evaluation suggests that the toolkit can serve as a valuable resource for practitioners designing PTG systems and researchers developing new tools for human-AI interaction design.

Figures (6)

• Figure 1: Low-fidelity prototyping allowed students to iterate on the design of MixITS systems with a small budget. Students created cardboard props representing real objects, such as a PC motherboard (a) for AI-assisted repair and kitchen utensils (c) for AI-assisted cooking. The MR interface prototypes used low-cost materials like paper labels to simulate virtual labels on a PC motherboard (a). Students moved these props to prototype dynamic behaviors, such as a virtual arrow pointing to parts of a PC (b) and tracking ingredients with bounding boxes made of straws(d), similar to low-fidelity paper prototyping of mobile or web applications. They also used inexpensive materials to represent customized hardware, including a haptic feedback glove (e) and a 360-degree camera headset (f). To simulate large outdoor environments, such as a four-way stop for an AI-supported navigation app for blind users, students created miniature versions using dioramas, figurines, and toys with extra sensors such as cameras embedded in paper hats. This approach allowed them to enact full system functionality in a controlled, scaled-down setting that represented complex real-world scenarios(f).
• Figure 2: Six MixITS Design Considerations produced by a reflexive thematic analysis of data collected in the design course. Section \ref{['sec:themes']} presents the considerations in detail.
• Figure 3: When applying Don Norman's Gulfs of Execution and Evaluation norman1986usernorman2013design to MixITS systems, we identify eight Gulfs that occur during interactions between the Human user, AI-MR system, and the Real Environment.
• Figure 4: The MixITS Interaction Canvas can help designers identify interaction problems leading to the Gulfs of Execution and Evaluation between the AI system, Human users, and the Environment. Designers can use this visual tool by filling out the blanks, in gray text for both the questions and the gulfs.
• Figure 5: (a) In Task 1, participants used Interaction Canvases to identify a Gulf on fictional-user issues. (b) They then referred to Table \ref{['tab:full']} to find a problem that most closely matched their scenario and identified the applicable Design Pattern. (c) In Task 3, they revisited their initial solution through the lens of a Design Consideration. P6 used descriptive AR labels (H-Ex-E-1) with didactic instructions on the importance of crashpads and their use, considering "Teaching and Directing". After reflecting on "Build Trust", P10 added an AI explanation to the goal inferred by AI (AI-Ev-H-36).