SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation

TL;DR

This work defines multi-channel sign language translation (MCSLT) and proposes SignBLEU, a BLEU-inspired metric that evaluates concurrent manual and non-manual signing signals. By introducing blockification and linearization, SignBLEU computes temporal and channel n-grams from multi-channel block data, enabling robust evaluation across three diverse sign-language corpora. Experimental results show SignBLEU consistently correlates better with human judgments than traditional SLT metrics, and the authors provide initial benchmark scores and open-source code to spur further MCSLT research. The approach offers a practical, scalable way to assess richer, multi-channel sign language translations and sets the stage for more faithful, information-rich SLT systems.

Abstract

Sign languages are multi-channel languages that communicate information through not just the hands (manual signals) but also facial expressions and upper body movements (non-manual signals). However, since automatic sign language translation is usually performed by generating a single sequence of glosses, researchers eschew non-manual and co-occurring manual signals in favor of a simplified list of manual glosses. This can lead to significant information loss and ambiguity. In this paper, we introduce a new task named multi-channel sign language translation (MCSLT) and present a novel metric, SignBLEU, designed to capture multiple signal channels. We validated SignBLEU on a system-level task using three sign language corpora with varied linguistic structures and transcription methodologies and examined its correlation with human judgment through two segment-level tasks. We found that SignBLEU consistently correlates better with human judgment than competing metrics. To facilitate further MCSLT research, we report benchmark scores for the three sign language corpora and release the source code for SignBLEU at https://github.com/eq4all-projects/SignBLEU.

Figures (6)

• Figure 1: Comparison of SCSLT and MCSLT ("d": dominant hand, "nd": non-dominant hand, and "nms": non-manual signal).
• Figure 2: The number of annotated channels of published sign language corpora by year.
• Figure 3: An example of blockification and linearization.
• Figure 4: The average $gram_t$ and $gram_c$ counts per sentence by maximum gram order.
• Figure 5: Correlations with human judgments. Highest scores are highlighted in bold for readability.