Voicing Uncertainty: How Speech, Text, and Visualizations Influence Decisions with Data Uncertainty

TL;DR

The paper addresses how data uncertainty is communicated via visual, textual, and spoken modalities, examining the role of acoustic factors such as pitch and accent in shaping decisions, confidence, and trust. It uses a crowdsourced, between-subjects design with 275 participants and six speech voices to compare unimodal and bimodal uncertainty representations. Key findings show that speech-forward representations increase risky decisions, text-forward representations reduce decision confidence, and trust is not consistently higher for speech in a bimodal setup; pitch can modestly boost confidence, and accented voices may affect trust. The work extends prior findings by incorporating multiple voices and accents, highlighting the importance of acoustic and contextual factors in designing effective uncertainty communication tools with practical implications for dashboards, voice assistants, and policy communications.

Abstract

Understanding and communicating data uncertainty is crucial for informed decision-making across various domains, including finance, healthcare, and public policy. This study investigates the impact of gender and acoustic variables on decision-making, confidence, and trust through a crowdsourced experiment. We compared visualization-only representations of uncertainty to text-forward and speech-forward bimodal representations, including multiple synthetic voices across gender. Speech-forward representations led to an increase in risky decisions, and text-forward representations led to lower confidence. Contrary to prior work, speech-forward forecasts did not receive higher ratings of trust. Higher normalized pitch led to a slight increase in decision confidence, but other voice characteristics had minimal impact on decisions and trust. An exploratory analysis of accented speech showed consistent results with the main experiment and additionally indicated lower trust ratings for information presented in Indian and Kenyan accents. The results underscore the importance of considering acoustic and contextual factors in presentation of data uncertainty.

Figures (5)

• Figure 1: Example stimuli viewed by participants. (a) Visualization-only representation. (b) Text-forward representation. The complete text template can be found in \ref{['sec:text_template']}. (c) Speech-forward representation. Descriptions and links for speech-forward conditions can be found in \ref{['sec:speech_links']}. The example in this image can be found https://osf.io/kxqfw.
• Figure 2: Proportion of decision types for each forecast. Overall, decisions were mostly rational. Speech-forward was the least rational representation, with a greater proportion of risky decisions. This observation was true across different voices as well.
• Figure 3: Confidence ratings ranged from 50 to 100. Overall, confidence was lower for text-forward ($mean = 82.4$) than for speech-forward forecasts ($mean = 85.5$).
• Figure 4: Average trust ratings. There were no significant differences between modes, but ratings were higher overall for speech-forward forecasts ($mean = 6.7$) than for text-forward ($mean = 6.3$) or traditional visualization ($mean = 6.1$).
• Figure 5: Findings from an exploratory investigation of accents. (a) Decision rationality between different voices tested. Risky decisions were consistently more common for speech-forward forecasts but did not vary by accent. (b) Confidence ratings and normalized pitch. Accent variants continued the minor trend observed in the experiment; there were small gains in confidence for increases in pitch. (c) Trust in forecast by accent and gender. Indian and Kenyan accents tended to have the lowest average trust.