Multimodality in VR: A survey

TL;DR

This paper surveys multimodality in virtual reality, focusing on how visual, auditory, haptic, proprioceptive, olfactory, and gustatory cues interact to shape realism, presence, attention, and performance. It organizes the literature around fidelity/presence, attention, task performance, perceptual illusions, navigation, and applications in medicine, education, and entertainment, highlighting concrete benefits as well as design challenges. The key contributions include a synthesis of crossmodal effects, guidelines for balancing modalities, and identification of gaps in empirical knowledge, hardware constraints, and data-driven modeling needs. The work provides a valuable, theory-informed compass for researchers and practitioners aiming to design richer, more effective multimodal VR experiences with realistic user embodiment and improved transfer to real-world tasks.

Abstract

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer.

Figures (10)

• Figure 1: VR can be used to systematically analyze the interactions of multimodal information. In this example, the influence of auditory signals in the perception of visual motion is studied malpica2020crossmodal. The authors found that different temporal synchronization profiles affected how the stimuli were perceived: When the visual (red balls moving) and auditory (an impact sound) stimuli were correctly synchronized, users perceived a unified event, in particular a collision between both balls.
• Figure 2: Structure of this state-of-the-art report. We divide it into different areas of the VR experience in which multimodality can play a key role: user immersion, presence, and realism of the experience (Section \ref{['sec:n3_fidelity']}); user attention when exploring the virtual environment (Section \ref{['sec:n5_guidance']}); user performance when completing tasks (Section \ref{['sec:n6_performance']}); multimodal perceptual illusions that can be leveraged in VR (Section \ref{['sec:n7_manipulation']}); and navigational effects of multimodality in virtual environments (Section \ref{['sec:new_7_nav']}). Finally, we review different applications where multimodality has been shown to improve the end goal (Section \ref{['sec:n8_applications']}), and finalize with a discussion on the need for multimodality, and open avenues of research (Section \ref{['sec:n9_conclusions']}).
• Figure 3: Including correct and coherent auditory information in the virtual environment has been proved to increase realism and immersion. Left: A system that automatically generates ambisonic information that creates a smoother acoustic experience for the scene morgado2018self. Right: A framework to include auditory information into 360º panoramas depending on the elements in the scene huang2019audible. In both cases, their validation experiments yield users' preference when auditory information is included, and an overall increase in the perceived realism and immersion.
• Figure 4: Left: Synchronizing different modalities increases the feeling of presence and the perception of the self. Moreover, multimodality can even create a distortion of that perception: Normand et al. normand2011multisensory presented a study where a body distortion illusion is achieved by synchronous visual-tactile and visual-motor correlations. Right: Some works have studied how different physical and behavioral factors can directly affect, and even manipulate, embodiment neyret2020embodied, and thereofre, the perception of the self.
• Figure 5: Saliency maps show the likelihood of users directing their attention to each part of the scene. Most of the current literature has been devoted to estimating saliency in unimodal, visual stimuli. This image shows the recent visual saliency estimation method proposed by Martin et al. martin20saliency (Left: Input panorama. Right: Estimated saliency). It has been shown that each sensory modality has the potential of influencing users' attentional behavior, therefore, there is a need for further exploration of multimodal saliency in VR.