Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches

Pengcheng An, Jiawen Zhu, Zibo Zhang, Yifei Yin, Qingyuan Ma, Che Yan, Linghao Du, Jian Zhao

TL;DR

EmoWear introduces an animated, pre-retrieval emotional teaser system for smartwatch voice messages, using a multimodal fusion model to surface two likely emotions from semantic and acoustic cues. Through a within-group comparison with a color-coded baseline (N=24), EmoWear improves receivers' pre-retrieval emotion interpretation, enhances senders’ emotional expression, and enriches the overall communication experience, while remaining approachable and engaging. The work provides a detailed account of design considerations, animation-based teaser design, and a multi-modal emotion classification framework, along with qualitative insights and actionable design implications for future HCI research on emotional teasers. Overall, EmoWear demonstrates the feasibility and value of glanceable, animated emotional cues to augment asynchronous voice communication on wearable devices, with broad implications for inclusivity, customization, and multimodal interaction.

Abstract

Voice messages, by nature, prevent users from gauging the emotional tone without fully diving into the audio content. This hinders the shared emotional experience at the pre-retrieval stage. Research scarcely explored "Emotional Teasers"-pre-retrieval cues offering a glimpse into an awaiting message's emotional tone without disclosing its content. We introduce EmoWear, a smartwatch voice messaging system enabling users to apply 30 animation teasers on message bubbles to reflect emotions. EmoWear eases senders' choice by prioritizing emotions based on semantic and acoustic processing. EmoWear was evaluated in comparison with a mirroring system using color-coded message bubbles as emotional cues (N=24). Results showed EmoWear significantly enhanced emotional communication experience in both receiving and sending messages. The animated teasers were considered intuitive and valued for diverse expressions. Desirable interaction qualities and practical implications are distilled for future design. We thereby contribute both a novel system and empirical knowledge concerning emotional teasers for voice messaging.

EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches

TL;DR

EmoWear introduces an animated, pre-retrieval emotional teaser system for smartwatch voice messages, using a multimodal fusion model to surface two likely emotions from semantic and acoustic cues. Through a within-group comparison with a color-coded baseline (N=24), EmoWear improves receivers' pre-retrieval emotion interpretation, enhances senders’ emotional expression, and enriches the overall communication experience, while remaining approachable and engaging. The work provides a detailed account of design considerations, animation-based teaser design, and a multi-modal emotion classification framework, along with qualitative insights and actionable design implications for future HCI research on emotional teasers. Overall, EmoWear demonstrates the feasibility and value of glanceable, animated emotional cues to augment asynchronous voice communication on wearable devices, with broad implications for inclusivity, customization, and multimodal interaction.

Abstract

Voice messages, by nature, prevent users from gauging the emotional tone without fully diving into the audio content. This hinders the shared emotional experience at the pre-retrieval stage. Research scarcely explored "Emotional Teasers"-pre-retrieval cues offering a glimpse into an awaiting message's emotional tone without disclosing its content. We introduce EmoWear, a smartwatch voice messaging system enabling users to apply 30 animation teasers on message bubbles to reflect emotions. EmoWear eases senders' choice by prioritizing emotions based on semantic and acoustic processing. EmoWear was evaluated in comparison with a mirroring system using color-coded message bubbles as emotional cues (N=24). Results showed EmoWear significantly enhanced emotional communication experience in both receiving and sending messages. The animated teasers were considered intuitive and valued for diverse expressions. Desirable interaction qualities and practical implications are distilled for future design. We thereby contribute both a novel system and empirical knowledge concerning emotional teasers for voice messaging.
Paper Structure (34 sections, 1 equation, 4 figures, 2 tables)

This paper contains 34 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Affective Enhancement in Text Messaging
  4. Affective Aid in Audio Transcription
  5. Mobile Voice Message Interactions
  6. Social-Emotional Interactions on Smartwatches
  7. EmoWear Interface Design
  8. Design Considerations
  9. EmoWear Interface and Usage Scenario
  10. Message Bubble Animations
  11. EmoWear System Implementation
  12. Emotion Classification Algorithm
  13. Speech-to-Emotion model
  14. Text-to-Emotion model
  15. Multi-modal fusion
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Seven examples from the 30 emotional teaser animations of EmoWear.
  • Figure 2: Overview of the EmoWear system architecture
  • Figure 3: (a,b,c) the EmoWear system; (d,e,f) the Baseline system implemented for comparison; (a,d) message receiving/recording; (b,e) an example emotional teaser of anger; (c,f) an example emotional teaser of sadness. In EmoWear, each emotion has five distinct animations, while in the Baseline system, each emotion has five color variations (differing in brightness).
  • Figure 4: User perceptions of EmoWear and Baseline on $7$-point Likert scales ($1=$ Strongly Disagree, $7=$ Strongly Agree, $4=$ Neutral; bar lengths represent medians, error bars represent $95\%$ CI by bootstrapping; * $p<=.05$; **$p<=.01$).