Perspectives on Capturing Emotional Expressiveness in Sign Language

TL;DR

This study investigates how emotions are communicated in sign languages through eight semi-structured interviews across three countries, revealing that emotional expressiveness is multimodal and embedded in grammar, with universal and culturally specific patterns. It critically examines current sign-language technologies, highlighting gaps in capturing affective nuance and the challenges of translation and interpretation across contexts. The authors propose design principles—feedback loops, context-aware deployment, visual accessibility, and Deaf-community co-design—to develop emotionally aware sign-language interfaces. The work advances theoretical understanding of emotional expression in sign language and offers practical guidance for building more inclusive, effective SL technologies with real-world impact in healthcare, education, and media.

Abstract

Significant advances have been made in our ability to understand and generate emotionally expressive content such as text and speech, yet comparable progress in sign language technologies remain limited. While computational approaches to sign language translation have focused on capturing lexical content, the emotional dimensions of sign language communication remain largely unexplored. Through semi-structured interviews with eight sign language users across Singapore, Sri Lanka and the United States, including both Deaf and Hard of hearing (DHH) and hearing signers, we investigate how emotions are expressed and perceived in sign languages. Our findings highlight the role of both manual and non-manual elements in emotional expression, revealing universal patterns as well as individual and cultural variations in how signers communicate emotions. We identify key challenges in capturing emotional nuance for sign language translation, and propose design considerations for developing more emotionally-aware sign language technologies. This work contributes to both theoretical understanding of emotional expression in sign language and practical development of interfaces to better serve diverse signing communities.