Design Space of Visual Feedforward And Corrective Feedback in XR-Based Motion Guidance Systems

TL;DR

This work maps a comprehensive design space for XR-based motion guidance by separately analyzing motion feedforward and corrective feedback, then examining their interaction. Grounded in a corpus of 38 papers, it defines four dimensions for feedforward (level of indirection, update strategy, viewing perspective, contextual cues) and four for corrective feedback (information level, temporality, placement, presentation), identifying nine feedforward configurations and multiple feedback presentation patterns. The framework is demonstrated through scenarios on sign language and deadlifting, and is extended with discussions of motion features, scoring, individuality, and avatar appearance. The study highlights opportunities for richer visual cues and contextual data, while acknowledging context-specific constraints and the need for future work on prioritization among design factors and integration of biometric data.

Abstract

Extended reality (XR) technologies are highly suited in assisting individuals in learning motor skills and movements -- referred to as motion guidance. In motion guidance, the "feedforward" provides instructional cues of the motions that are to be performed, whereas the "feedback" provides cues which help correct mistakes and minimize errors. Designing synergistic feedforward and feedback is vital to providing an effective learning experience, but this interplay between the two has not yet been adequately explored. Based on a survey of the literature, we propose design space for both motion feedforward and corrective feedback in XR, and describe the interaction effects between them. We identify common design approaches of XR-based motion guidance found in our literature corpus, and discuss them through the lens of our design dimensions. We then discuss additional contextual factors and considerations that influence this design, together with future research opportunities for motion guidance in XR.

Figures (3)

• Figure 1: Design dimensions of motion feedforward. (a) level of indirection at explicit, implicit and abstract. (b) interactive update strategy at discrete, continuous and autonomous. (c) viewing perspective at first-, mirror- and third-person perspectives; in the latter two perspectives, the feedforward will show around the duplicate of the trainee's avatar. (d) additional contextual cues such as speed, punch force and muscle activation. For illustrative purposes, grey represents the trainee's avatar in an egocentric perspective, blue represents feedforward instructions, green represents the completed part in implicit guidance, yellow represents the dynamic duplicate of trainee's avatar in mirror- or third-person perspective, and orange represents severity of speed, punch force, or muscle activation.
• Figure 2: Diagrammatic overview of how our design dimensions of level of indirection and interactive update strategy for feedforward relate to the original classifications by Wang et al. wang2022survey and Elsayed et al. elsayed2022understanding.
• Figure 3: Examples of corrective feedback in the literature: (a) Color: both the trainee's avatar and the virtual trainer are represented as ball-and-stick models, changing to green when corresponding body parts align, or to red and cyan when they do not align hoang2016onebody (courtesy of Vetere, © ACM). (b) Direction: a red arrow is positioned on the trainee's shoulder (in white), and provides a rectification feedback by pointing at the virtual coach's (blue) shoulder oshita2018self (courtesy of Oshita, © IEEE). (c) Size: the length of cyan lines indicates the deviation distance between the trainee's wrist and the desired posture yu2020perspective (courtesy of Yu, © IEEE). (d) Text: the detection of the alignment status of each body joint attached to a wall caserman2021full (courtesy of Caserman, © IEEE). (e) Graph: the graph on the bottom-left corner presents the magnitude of the movement error throughout the entire training phase escalona2020eva (courtesy of Escalona, © Springer).