A Hong Kong Sign Language Corpus Collected from Sign-interpreted TV News

TL;DR

TVB-HKSL-News introduces a large, signer-dependent HKSL dataset collected from TVB News to advance continuous SLR and SLT research for HKSL. The dataset provides 16.07 hours of sign video from two signers with 6,515 glosses (SLR) and 2,850 Chinese tokens (SLT), plus an automated collection and glossing pipeline. Baseline experiments with state-of-the-art SLR/SLT models establish benchmarks (WER ≈ 34.08% for SLR; BLEU-4 ≈ 23.58 for SLT) and reveal insights on data quantity and signer dependence. The work also details an end-to-end pipeline for extracting sign and subtitle content, annotating glosses, and enabling scalable future collection for HKSL and other sign languages, with ethical considerations and controlled dataset availability.

Abstract

This paper introduces TVB-HKSL-News, a new Hong Kong sign language (HKSL) dataset collected from a TV news program over a period of 7 months. The dataset is collected to enrich resources for HKSL and support research in large-vocabulary continuous sign language recognition (SLR) and translation (SLT). It consists of 16.07 hours of sign videos of two signers with a vocabulary of 6,515 glosses (for SLR) and 2,850 Chinese characters or 18K Chinese words (for SLT). One signer has 11.66 hours of sign videos and the other has 4.41 hours. One objective in building the dataset is to support the investigation of how well large-vocabulary continuous sign language recognition/translation can be done for a single signer given a (relatively) large amount of his/her training data, which could potentially lead to the development of new modeling methods. Besides, most parts of the data collection pipeline are automated with little human intervention; we believe that our collection method can be scaled up to collect more sign language data easily for SLT in the future for any sign languages if such sign-interpreted videos are available. We also run a SOTA SLR/SLT model on the dataset and get a baseline SLR word error rate of 34.08% and a baseline SLT BLEU-4 score of 23.58 for benchmarking future research on the dataset.

Figures (6)

• Figure 1: In the "TVB News Report with Sign Language" program, a news anchor speaks on the left side of the screen while an HKSL interpreter signs on the right. Chinese subtitles are displayed at the bottom of the screen.
• Figure 2: Keypoints extracted from Signer-1/Signer-2 in the first/second row, respectively.
• Figure 3: Pie charts showing statistics: number of samples, duration (hours), and running glosses/characters for SLR/SLT tasks across train/dev/test sets.
• Figure 4: The first three figures show the distribution of clip lengths, number of glosses per clip for SLR, and characters per clip for SLT. The last two figures show the occurrences of particular sign glosses and characters among the samples, sorted by their overall occurrences. Different colors represent different dataset splits.
• Figure 5: Example of subtitle background removal. Lines 1 and 2 show an example of sample input and target image used for training the U-Net model. The trained model processes Line 3 (input) to produce Line 4 (output) with background removed.