Two Empirical Studies on Audiovisual Semiotics of Uncertainty

TL;DR

This work tackles the lack of theory guiding audiovisual representations of uncertainty by conducting two preregistered crowd-sourced studies to 1) identify preferred audio-visual (AV) pairs and their polarity, and 2) map these AV stimuli to probability as a proxy for uncertainty. The findings reveal strong cross-modal preferences for certain AV pairs and demonstrate that not all preferred pairs serve well as uncertainty mappings, underscoring the need to couple AV channel choices with explicit data-to-channel mappings. The paper offers design guidelines for AV representations of uncertainty, emphasizing high-preference pairings and polarity-consistent mappings to minimize reaction time and maximize user preference. Overall, the results advance AV analytics byProviding empirical guidance for integrating sonification and visualization to enhance perception of uncertainty in scientific and applied contexts.

Abstract

There exists limited theoretical guidance on integrating visualization and sonification. In this paper, we address this gap by investigating audiovisual semiotics for uncertainty representation: joining uncertainty visualization and sonification to combine audiovisual channels for enhancing users' perception of uncertainty. We conducted two preregistered crowd-sourced user studies. First, we assessed suitable audio/visual pairs. Then, we investigated audiovisual mappings of uncertainty. Here, we use probability as it is an easily communicated aspect of uncertainty. We analyzed the participants' preferences and reaction times in both user studies. Additionally, we explored the strategies employed by participants through qualitative analysis. Our results reveal audiovisual mappings that lead to particularly strong preferences and low reaction times. Furthermore, we found that preferred audio/visual pairs are not necessarily suitable audiovisual mappings of uncertainty. For example, while pitch paired with brightness was preferred as a pair, it was not well suited as a mapping for uncertainty. We recommend audiovisual mappings of uncertainty that lead to low reaction times and high preferences in both user studies. This paper presents guidelines to anyone seeking to employ audiovisual representations for uncertainty, contributing to enhancing the perception of uncertainty.

Figures (13)

• Figure 1: The visual stimuli used in both experiments. The arrows indicate the polarities examined in the experiments for each channel. Recreated from MacEachren et al. maceachren2012visual.
• Figure 2: A textual and visual description of the audio channels used in both experiments. The arrows under polarity and visual description indicate the polarities per channel. The figure was adapted from Ballatore et al. ballatore2019sonifying.
• Figure 3: The order of trials in Experiment 1. Participants were first shown two visual stimuli. The visual stimuli were variants of the same channel (e.g., big-sized and small-sized circles). Once ready, participants pressed a button to continue, which removed the visual stimuli. Participants then listened to two audio stimuli separated by a 0.3-second interval. The audio stimuli were variants of the same channel (e.g., high pitch and low pitch). The visuals were then displayed again, and participants selected which visual better represented the second sound they heard.
• Figure 4: The mean proportion per visual and audio condition for each polarity preference. The error bars show the 95% confidence intervals. The further away from the reference line (binomial probability of 0.5), the stronger the preference is for that AV pair.
• Figure 5: The mean RT in milliseconds per visual and audio channel. The error bars represent the 95% CI. The lower the RT, the quicker the participants were to indicate their preference.