Embedded vs. Situated: An Evaluation of AR Facial Training Feedback

TL;DR

This paper investigates how the spatial placement of AR feedback influences facial motor training by comparing embedded (ARSelfie), proxy-embedded (Mannequin), and situated (BarChart) visualizations against a Baseline. Using a within-subjects study with $N=24$, it demonstrates that embedded feedback reduces extraneous cognitive load and enhances user experience and preference, while situated feedback yields better accuracy but incurs higher cognitive demand. The results support extending embedded feedback principles from gross-motor to facial training and highlight design trade-offs related to self-representation and perceptual clarity. The findings inform design guidelines for AR-based facial rehabilitation and performance training, balancing interpretability, comfort, and motor-learning efficacy.

Abstract

While augmented reality (AR) research demonstrates benefits of embedded visualizations for gross motor training, its applicability to facial exercises remains under-explored. Providing effective real-time feedback for facial muscle training presents unique design challenges, given the complexity of facial musculature. We developed three AR feedback approaches varying in spatial relationship to the user: situated (screen-fixed), proxy-embedded (on a mannequin), and fully embedded (overlaid on the user's face). In a within-subjects study (N=24), we measured exercise accuracy, cognitive load, and user preference during facial training tasks. The embedded feedback reduced cognitive load and received higher preference ratings, while the situated feedback enabled more precise corrections and higher accuracy. Qualitative analysis revealed a key design tension: embedded feedback improved experience but created self-consciousness and interpretive difficulty. We distill these insights into design considerations addressing the trade-offs for facial training systems, with implications for rehabilitation, performance training, and motor skill acquisition.

Figures (12)

• Figure 1: The facial muscle anatomy chart (a) (from nestor2017) that we used to map subsets of the 468 facial landmarks 3D face mesh (b) provided by Google ARCore AugmentedFaces API to the target muscles in our study.
• Figure 2: The activation estimation process, demonstrated with the left forehead muscle in the BarChart condition. First, the phone's front camera captures the user's face and the AugmentedFaces API returns a face mesh. The landmarks corresponding to a particular muscle are used to estimate its current position. This position along with the muscle's baseline and peak positions are used to compute a raw activation score. This score is mapped to an activation label and both are used to provide feedback to the user.
• Figure 3: The four experimental conditions used in the study, showing a participant performing the three facial exercises : (a) Baseline (Eyebrow Raise), (b) BarChart (Reverse Frown), (c) Mannequin (Eyebrow Raise) and (d) ARSelfie (Smile).
• Figure 4: Opacity-modulated feedback in the embedded conditions, illustrated for the forehead muscle on the ARCore face mesh. Color indicates the activation zone: blue for under-activation, green for the optimal range, and orange for over-activation. Opacity encodes distance from the optimal range: the blue overlay fades as activation approaches green, the green overlay is most opaque at the midpoint and more transparent near its lower and upper thresholds, and the orange overlay increases in opacity as activation rises further beyond optimal.
• Figure 5: Study procedure. After providing informed consent and completing the ATI questionnaire franke2019personal, participants received a tutorial on exercises, study task and feedback conditions through a presentation, after which they switched to the system for the calibration and exercise phases. In the calibration phase participants first maintained a neutral expression (10 seconds) and then performed the three exercises at maximum effort (10 seconds each with 5-second breaks). In the exercise phase, participants completed four counter-balanced trials (Latin-square design), one per condition. Each trial consisted of six repetitions of the exercise cycle (three exercises performed for 10 seconds in randomized order with 5-second breaks). After each trial, participants completed the NASA-TLX hart_development_1988, UEQ schrepp2017construction, and a custom extraneous cognitive load questionnaire.