SCOPE: Sign Language Contextual Processing with Embedding from LLMs

TL;DR

This work tackles the lack of contextual understanding in vision-based sign language processing by introducing SCOPE, a context-aware SLR and SLT framework that leverages embeddings aligned to a frozen LLM and fine-tunes an LLM with prior context. It introduces the SCOPE dataset—a large-scale Chinese sign language dialogue collection with gloss and translation annotations—and demonstrates state-of-the-art performance on multiple benchmarks (Phoenix-2014T, CSL-Daily, and SCOPE) while offering open-source code and data. The method combines an Embedding Alignment Encoder to align motion Features with text embeddings, a Gloss Embedding encoder that incorporates preceding contextual text, CTC decoding with MWER training, and Q-LoRA-based LLM fine-tuning to translate glosses into natural language under context. The results, together with user studies and a real-time translation demo in healthcare settings, highlight the practical impact of context-aware sign language processing for Deaf communities and broader accessibility research.

Abstract

Sign languages, used by around 70 million Deaf individuals globally, are visual languages that convey visual and contextual information. Current methods in vision-based sign language recognition (SLR) and translation (SLT) struggle with dialogue scenes due to limited dataset diversity and the neglect of contextually relevant information. To address these challenges, we introduce SCOPE (Sign language Contextual Processing with Embedding from LLMs), a novel context-aware vision-based SLR and SLT framework. For SLR, we utilize dialogue contexts through a multi-modal encoder to enhance gloss-level recognition. For subsequent SLT, we further fine-tune a Large Language Model (LLM) by incorporating prior conversational context. We also contribute a new sign language dataset that contains 72 hours of Chinese sign language videos in contextual dialogues across various scenarios. Experimental results demonstrate that our SCOPE framework achieves state-of-the-art performance on multiple datasets, including Phoenix-2014T, CSL-Daily, and our SCOPE dataset. Moreover, surveys conducted with participants from the Deaf community further validate the robustness and effectiveness of our approach in real-world applications. Both our dataset and code will be open-sourced to facilitate further research.

Figures (7)

• Figure 1: (a) Our SCOPE dataset contains rich contextual information and sign language videos. (b) Our SCOPE framework is a robust context-aware sign language recognition/translation model capable of recognizing dialogue-based sign language gestures, predicting glosses, and generating spoken sentences with the aid of LLMs.
• Figure 2: Overview of SCOPE framework. Our Embedding Alignment Encoder captures holistic linguistic information from the whole motion sequence. Aligning embedding space to match a frozen LLM enables integrating previous context information for SLR. Finally, Q-LoRA fine-tuning fits an LLM for translating predicted glosses with context into spoken language.
• Figure 3: SCOPE dataset collection pipeline. Given dialogue texts, experienced signers produce corresponding sign videos along with self-annotated glosses. For each video, other signers replicate data based on the glosses and the text.
• Figure 4: SCOPE gallery. We sampled different scenarios and show case the dataset sign videos and annotations.
• Figure 5: Comparison of 2D keypoints identified by MediaPipe (up) and DWPose (down) from the Phoenix 2014 dataset. DWPose provides more accurate and detailed keypoints.