SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness

TL;DR

This work addresses how to make mixed-reality soundscapes accessible to blind and visually impaired users by introducing SoundShift, a framework of six sound manipulators that balance real-world and virtual audio. Grounded in a content-analysis of online forums, the authors design six manipulations and evaluate them in Unity across three MR scenarios with 18 BVI participants, comparing against full transparency and noise cancellation. The study demonstrates that SoundShift improves sound awareness and reduces cognitive load, with scenario-specific trade-offs and strong user feedback for customization. Three real-world prototype applications illustrate SoundShift's practicality and potential to inform future MR accessibility design.

Abstract

Mixed-reality (MR) soundscapes blend real-world sound with virtual audio from hearing devices, presenting intricate auditory information that is hard to discern and differentiate. This is particularly challenging for blind or visually impaired individuals, who rely on sounds and descriptions in their everyday lives. To understand how complex audio information is consumed, we analyzed online forum posts within the blind community, identifying prevailing challenges, needs, and desired solutions. We synthesized the results and propose SoundShift for increasing MR sound awareness, which includes six sound manipulations: Transparency Shift, Envelope Shift, Position Shift, Style Shift, Time Shift, and Sound Append. To evaluate the effectiveness of SoundShift, we conducted a user study with 18 blind participants across three simulated MR scenarios, where participants identified specific sounds within intricate soundscapes. We found that SoundShift increased MR sound awareness and minimized cognitive load. Finally, we developed three real-world example applications to demonstrate the practicality of SoundShift.

Figures (10)

• Figure 1: Simulated RW-Focused Scenario. (a) The user's avatar in Unity wears a headphone and holds a white cane. (b) The user sets the sound output by placing audio directions on the left, ringtone on the right, and (c) navigates on the street. (d) Several crowds, vendors, and passing cars along the street generate ambient noises, as well as (e) construction sites with drilling noises. (f) While walking, the user might come across random manholes, causing the white cane's sound to change upon contact.
• Figure 2: Simulated VR-Focused Scenario. (a) The user’s avatar in Unity wears headphones, sits and works at the help desk, listens to an audio handbook, and (b) occasionally receives voice notes from a supervisor. (c) In the environment, there are background noises when people walk around, talk to each other, and open/close the sliding door. (d) People sometimes knock on the desk to get the user's attention. (e) The speaker plays occasional public announcements on the front wall.
• Figure 3: Simulated Fully-Mixed Scenario. (a) The user's avatar sits at the dining table and wears headphones to consume the voice of virtual speakers and virtual broadcasts. (b) There are physical and remote virtual speakers on the front stage, (c) where they may stand and speak up at the same time in a panel discussion, (d) similar to the physical and virtual attendees around the table. (e) Waitstaff sometimes comes to clean the table, generating dish clinking sounds.
• Figure 4: In our study, participants wore headphones, engaged in pre-defined scenarios, and pressed specific keys upon hearing corresponding sounds.
• Figure 5: Results based on Condition (RQ1) or Scenario (RQ2). **=$p$<=0.001. ***=$p$<0.0001.