How Pragmatics Shape Articulation: A Computational Case Study in STEM ASL Discourse

TL;DR

This study investigates how pragmatics shape sign articulation in ASL STEM discourse by collecting a motion-capture dataset that compares dyadic dialogue, solo lectures, and interpreted articles. It introduces a set of kinematic metrics for spatial, temporal, and vertical articulation, and demonstrates reliable signs of dialogue-driven reductions absent in monologue contexts. Embedding-based analyses reveal that current sign-language models struggle to generalize across pragmatic variation, highlighting a gap between linguistic adaptation and machine representations. The work provides an empirical bridge between sign-language pragmatics and computational modeling, with implications for more robust, discourse-aware sign-language technologies and education tools.

Abstract

Most state-of-the-art sign language models are trained on interpreter or isolated vocabulary data, which overlooks the variability that characterizes natural dialogue. However, human communication dynamically adapts to contexts and interlocutors through spatiotemporal changes and articulation style. This specifically manifests itself in educational settings, where novel vocabularies are used by teachers, and students. To address this gap, we collect a motion capture dataset of American Sign Language (ASL) STEM (Science, Technology, Engineering, and Mathematics) dialogue that enables quantitative comparison between dyadic interactive signing, solo signed lecture, and interpreted articles. Using continuous kinematic features, we disentangle dialogue-specific entrainment from individual effort reduction and show spatiotemporal changes across repeated mentions of STEM terms. On average, dialogue signs are 24.6%-44.6% shorter in duration than the isolated signs, and show significant reductions absent in monologue contexts. Finally, we evaluate sign embedding models on their ability to recognize STEM signs and approximate how entrained the participants become over time. Our study bridges linguistic analysis and computational modeling to understand how pragmatics shape sign articulation and its representation in sign language technologies.

Figures (4)

• Figure 1: We explore the effects of pragmatics in three different signed STEM contexts: 1) instructor-student dyadic conversations, 2) isolated vocabulary, 3) a signed lecture, and 4) interpreted Wikipedia articles. We computationally analyze how signers establish conceptual pacts across these contexts.
• Figure 2: Sign duration patterns over time for instructor-student ASL dialogue. Stacked panels show start time (x) vs. duration (y) of individual signs for both participants, with connected lines indicating temporal progression of repeated signs within semantic groups. Each point represents an individual STEM sign instance, colored by semantic base term and shaped by articulation variation (circles, squares, diamonds for fingerspelling). Light blue background shading indicates active signing periods for each participant.
• Figure 3: Path length differences between dialogue and vocabulary articulation for left and right hands as a function of mention order. Colors correspond to individual signs and the bolded black line is the mean with standard error. A value of 0 indicates no difference between dialogue and vocabulary path lengths. Negative values indicate shorter trajectories in dialogue. Overall, dialogue signing shows consistently shorter and progressively reduced movement paths relative to isolated signs.
• Figure 4: Sign productions plotted with respect to their embeddings' L2 distance to the mean of each signer's average and the group's average.