EASLT: Emotion-Aware Sign Language Translation

TL;DR

EASLT tackles semantic ambiguity in gloss-free Sign Language Translation by elevating facial expressions from auxiliary cues to primary semantic anchors. It introduces a decoupled, multi-path architecture with dedicated encoders for spatial, motion, and emotion features ($\mathbf{Z}_s$, $\mathbf{Z}_m$, $\mathbf{Z}_e$) and a novel Emotion-Aware Fusion that modulates these streams before a short-term temporal layer feeds a language model. A multimodal alignment loss plus cross-entropy generation loss, optimized with LoRA, yields state-of-the-art results among gloss-free methods on PHOENIX14T ($B_4=26.15$, $\text{BLEURT}=61.0$) and CSL-Daily ($B_4=22.80$, $\text{BLEURT}=57.8$), outperforming prior approaches. The work demonstrates that explicit emotion modeling markedly improves translation fidelity and disambiguates signs whose meanings hinge on facial affect, with broad implications for sign-language technologies and multilingual SLT systems.

Abstract

Sign Language Translation (SLT) is a complex cross-modal task requiring the integration of Manual Signals (MS) and Non-Manual Signals (NMS). While recent gloss-free SLT methods have made strides in translating manual gestures, they frequently overlook the semantic criticality of facial expressions, resulting in ambiguity when distinct concepts share identical manual articulations. To address this, we present **EASLT** (**E**motion-**A**ware **S**ign **L**anguage **T**ranslation), a framework that treats facial affect not as auxiliary information, but as a robust semantic anchor. Unlike methods that relegate facial expressions to a secondary role, EASLT incorporates a dedicated emotional encoder to capture continuous affective dynamics. These representations are integrated via a novel *Emotion-Aware Fusion* (EAF) module, which adaptively recalibrates spatio-temporal sign features based on affective context to resolve semantic ambiguities. Extensive evaluations on the PHOENIX14T and CSL-Daily benchmarks demonstrate that EASLT establishes advanced performance among gloss-free methods, achieving BLEU-4 scores of 26.15 and 22.80, and BLEURT scores of 61.0 and 57.8, respectively. Ablation studies confirm that explicitly modeling emotion effectively decouples affective semantics from manual dynamics, significantly enhancing translation fidelity. Code is available at https://github.com/TuGuobin/EASLT.

Figures (5)

• Figure 1: Resolving semantic ambiguity through NMS: Examples from CSL-Daily zhou2021improving where "REVENGE" and "RECIPROCATE" exhibit nearly identical MS but are disambiguated by contrasting facial expressions. Note: To adhere to ethical guidelines and privacy standards, portrait visualizations are stylized using Nano Banana Pro google2025nano.
• Figure 2: Overview of the proposed EASLT framework. The pipeline consists of three sequential stages: (i) Multimodal Feature Extraction: Spatial, motion, and emotion features are extracted from preprocessed video inputs using specialized encoders. (ii) Emotion-Aware Fusion: This module comprises an Emotion-Aware Modulation that distills emotional cues to dynamically modulate spatial, motion, and emotion features, followed by a Temporal Layer (1D TCN and MLP) performing short-term modeling and projecting fused representations into the LLM's latent space. (iii) Translation Generation: The projected features are concatenated with text prompts to fine-tune the LLM via LoRA, utilizing contrastive learning for multimodel alignment.
• Figure 3: Detailed architecture of the EAM, comprising two key components: Enhancer that improves feature reliability through gating and quality assessment and Modulator that enables adaptive feature modulation using global emotion signals.
• Figure 4: Distribution of images across the seven emotion categories in the FER2013 dataset.
• Figure 5: Video segments from PHOENIX14T camgoz2018neural demonstrating almost identical MS for "HOTTER" and "SNOW", which are disambiguated exclusively by contrasting NMS.