Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning

TL;DR

A skill-adaptive, dynamically transparent ghost instructor whose opacity adapts in real time to learner performance is introduced, suggesting that adaptive transparency helps learners internalize fingerings more effectively, reducing dependency on external cues and improving short-term skill retention within immersive learning environments.

Abstract

Motor-skill learning systems in XR rely on persistent cues. However, constant cueing can induce overreliance and erode memorization and skill transfer. We introduce a skill-adaptive, dynamically transparent ghost instructor whose opacity adapts in real time to learner performance. In a first-person perspective, users observe a ghost hand executing piano fingering with either a static or a performance-adaptive transparency in a VR piano training application. We conducted a within-subjects study (N=30), where learners practiced with traditional Static (fixed-transparency) and our proposed Dynamic (performance-adaptive) modes and were tested without guidance immediately and after a 10-minute retention interval. Relative to Static, the Dynamic mode yielded higher pitch and fingering accuracy and limited error increases, with comparable timing. These findings suggest that adaptive transparency helps learners internalize fingerings more effectively, reducing dependency on external cues and improving short-term skill retention within immersive learning environments. We discuss design implications for motor-skill learning and outline directions for extending this approach to longer-term retention and more complex tasks.

Figures (9)

• Figure 1: System interface and dynamic-transparency feedback. The blue hand denotes the participant, and the gray hand denotes the ghost instructor. (a) Controls. Passthrough desk alignment; ghost audio mute/unmute; record button. The second row selects Phrase 1, Phrase 2, or Full Melody and shows the remaining-trial count (default 10). The Adjust panel fine-tunes the virtual piano rig to the physical desk (translate/rotate for alignment). (b--d) Adaptive feedback. Ghost-hand opacity is driven by performance in real time: higher visibility under poorer performance, and fainter visibility as pitch, timing, and fingering stabilize.
• Figure 2: Participant distribution across Latin-square conditions and piano experience. S = Static, D = Dynamic; each condition pairs a visibility mode with a melody A/B.
• Figure 3: Performance comparison between Static and Dynamic ghost hand conditions across pitch accuracy, finger accuracy, timing accuracy, and error rate. Error bars represent the standard error of the mean (SEM).
• Figure 4: Retention scores (Retention -- Immediate) for pitch, finger, and timing accuracy as well as error rate, comparing Dynamic and Static ghost hand conditions. Values near zero indicate stable performance, with negative values showing decline and positive values showing improvement.
• Figure 5: Interaction plots showing performance across Static and Dynamic conditions for experienced and non-experienced participants.