Effects of Multisensory Feedback on the Perception and Performance of Virtual Reality Hand-Retargeted Interaction

Abstract

Retargeting methods that modify the visual representation of real movements have been widely used to expand the interaction space and create engaging virtual reality experiences. For optimal user experience and performance, it is essential to specify the perception of retargeting and utilize the appropriate range of modification parameters. However, previous studies mostly concentrated on whether users perceived the target sense or not and rarely examined the perceptual accuracy and sensitivity to retargeting. Moreover, it is unknown how the perception and performance in hand-retargeted interactions are influenced by multisensory feedback. In this study, we used rigorous psychophysical methods to specify users' perceptual accuracy and sensitivity to hand-retargeting and provide acceptable ranges of retargeting parameters. We also presented different multisensory feedback simultaneously with the retargeting to probe its effect on users' perception and task performance. The experimental results showed that providing continuous multisensory feedback, proportionate to the distance between the virtual hand and the targeted destination, heightened the accuracy of users' perception of hand retargeting without altering their perceptual sensitivity. Furthermore, the utilization of multisensory feedback considerably improved the precision of task performance, particularly at lower gain factors. Based on these findings, we propose design guidelines and potential applications of VR hand-retargeted interactions and multisensory feedback for optimal user experience and performance.

Figures (8)

• Figure 1: The overall experiment procedure. Before a block of each experimental condition starts, participants go through the practice phase. During the experiment, participants perform a simple hand-reaching task while the hand is retargeted and answer the 2AFC question (slower or faster) regarding the speed of the virtual hand's movement compared to the real hand. Each block consists of 100 trials (20 repetitions for five gain factors), and participants respond to the embodiment questionnaire at the end of each block. This procedure is repeated for five experimental conditions with different multisensory feedback.
• Figure 2: A participant's first-person view in VR and third-person view of a participant during the hand-reaching task. (a) Participants started a trial by touching the green-colored origin point located near their torso with their right index finger. (b) Participants reached out their right hand to touch the target within an accessible area, and the target turned red when touched.
• Figure 3: A participant's right hand with all sensors attached. The vibration motor is fixed under the tip of the right index finger; the finger splint helps participants maintain a pointing gesture. The virtual hand moves according to the real hand's movement by the tracker on the back of the participant's hand.
• Figure 4: The result of perceptual features. (a) Psychometric functions derived from participants' 2AFC responses for each multisensory feedback condition. The x-coordinates for the intersecting points of the psychometric functions and the solid line are the PSEs (50%) and the two dashed lines are the upper (75%) and lower (25%) thresholds. (b) The PSEs and (c) the upper thresholds derived from the psychometric function for each multisensory feedback condition. The PSE and the upper threshold showed analogous statistical differences between conditions, considering the marginally significant difference (p=.051) between the None and V conditions in the upper threshold. The dashed lines represent the gain factor of 1, where the VR hand moved identically to the real hand. The error bars represent 95% confidence intervals. *p$<$.05, **p$<$.01, ***p$<$.001.
• Figure 5: The offset of point of subjective equality (PSE) from the standard gain factor (1.0) for each multisensory feedback condition. The error bars represent 95% confidence intervals. *p$<$.05, **p$<$.01, ***p$<$.001.