The Ballad of the Bots: Sonification Using Cognitive Metaphor to Support Immersed Teleoperation of Robot Teams

TL;DR

This work compares two public-facing sonification design strategies for VR-based teleoperation of robot teams in a hazardous nuclear decommissioning scenario: a cognitive metaphor approach aligned with embodied experience and an established computationalist mapping. Using a simulated digital twin in VR, the study conducts a between-subjects user study and an online hazard-identification task to evaluate presence, trust, workload, and data interpretability. The main finding is that the computationalist design offers slightly better predictability and lower mental workload, while the cognitive metaphor design yields more intuitive, spatially legible sounds when multiple sources are present; results also indicate no strong gain in presence from either approach. The work highlights the importance of long-term exposure and potential hybrid designs to optimize both usability and perceptual effectiveness in VR-enabled robotic teams for hazardous environments.

Abstract

As an embodied and spatial medium, virtual reality is proving an attractive proposition for robot teleoperation in hazardous environments. This paper examines a nuclear decommissioning scenario in which a simulated team of semi-autonomous robots are used to characterise a chamber within a virtual nuclear facility. This study examines the potential utility and impact of sonification as a means of communicating salient operator data in such an environment. However, the question of what sound should be used and how it can be applied in different applications is far from resolved. This paper explores and compares two sonification design approaches. The first is inspired by the theory of cognitive metaphor to create sonifications that align with socially acquired contextual and ecological understanding of the application domain. The second adopts a computationalist approach using auditory mappings that are commonplace in the literature. The results suggest that the computationalist approach outperforms the cognitive metaphor approach in terms of predictability and mental workload. However, qualitative data analysis demonstrates that the cognitive metaphor approach resulted in sounds that were more intuitive, and were better implemented for spatialisation of data sources and data legibility when there was more than one sound source.

Figures (7)

• Figure 1: The virtual reality environment. Voxel objects that have been revealed by the robots can be seen and one is tagged as radioactive. The laser pointer is in radiation tagging mode. A robot is selected in waypoint mode (green outlined robot), and two waypoints have been set. A hazard marker is visible by the barrels on the right.
• Figure 2: Layout of the simulated nuclear environment. Hazard spheres are made visible and are red for temperature, green for radiation, and blue for flammable gas. The robots and the user avatar start in the top right-hand corner.
• Figure 3: The video interface used for the sound evaluation study. The circle traverses the horizontal scale displaying its position.
• Figure 4: Results distributions for the mean errors in data-level identification. Gas (G), Temperature (T) and Radiation (R). Significant comparisons are indicated with a *
• Figure 5: Frequency with which the data types were correctly identified. Gas (G), Temperature (T) and Radiation (R).