Hidden bawls, whispers, and yelps: can text be made to sound more than just its words?

Abstract

Whether a word was bawled, whispered, or yelped, captions will typically represent it in the same way. If they are your only way to access what is being said, subjective nuances expressed in the voice will be lost. Since so much of communication is carried by these nuances, we posit that if captions are to be used as an accurate representation of speech, embedding visual representations of paralinguistic qualities into captions could help readers use them to better understand speech beyond its mere textual content. This paper presents a model for processing vocal prosody (its loudness, pitch, and duration) and mapping it into visual dimensions of typography (respectively, font-weight, baseline shift, and letter-spacing), creating a visual representation of these lost vocal subtleties that can be embedded directly into the typographical form of text. An evaluation was carried out where participants were exposed to this speech-modulated typography and asked to match it to its originating audio, presented between similar alternatives. Participants (n=117) were able to correctly identify the original audios with an average accuracy of 65%, with no significant difference when showing them modulations as animated or static text. Additionally, participants' comments showed their mental models of speech-modulated typography varied widely.

Figures (7)

• Figure 1: Diagram of how the model combines the sound of a given spoken utterance (identified by the green 1) with its timed transcription (2), generating text that has visual cues for prosody (3). This processing is done in two parts: first, prosodic features are extracted from the audio and processed (A and B, discussed, respectively, in sections \ref{['subsec:feature_extraction']} and \ref{['subsec:feature_processing']}). Following, these values are used to modulate typographic parameters (C, discussed in section \ref{['subsec:modulation_of_typography']}).
• Figure 2: The three normalizations. From top to bottom, the one that considers the whole utterance, the one that considers a window of 15 positions around the current one, and the arithmetic mean of the two. To accentuate the effect for illustrative purposes, in this figure we have used random numbers applied to each letter --- rather than to each syllable, as was used in our model.
• Figure 3: The words 'modulated typography' are shown with a decomposition of how the model will modulate each syllable using the three typographic parameters used in our model. On the row identified by the number 1, the words are in their default state with the syllabic boundaries highlighted (for this example we are using the Inter typeface). On row 2, each of the three typographic modulations is represented, one in each line. They are, respectively, font-weight, letter-spacing, and baseline shift. Underneath each syllable, in pink, we marked the relative values each parameter received. On row 3, the finished product, with the three typographic modulations applied at the same time.
• Figure 4: Example of how a single glyph changes shape between two extremes of an axis of variation. In this case, we are showing how the font-weight axis changes the letter e from the Inter typeface, going from the lighter value of 200 (in pink) to the bolder value of 900 (in blue).
• Figure 5: A translated screenshot of one round of the test. Participants had to match one of the audio files below with the video above, which was muted. Both audio files had a reading of the same text as presented in the video, but with different prosodic emphases. The words on the video say, in Portuguese, 'and when one least expects it / everything ends suddenly, / as does reality.' britto2003macau