Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media

TL;DR

This work tackles the data bottleneck in sign-language translation by introducing a Vision-Language Model–assisted pipeline for automated data collection, filtering, annotation, and verification from social media. It uses Qwen-2.5-VL for face visibility, signing activity, and OCR-based text extraction, with Phi-4-multimodal as a model-for-verification to ensure alignment between signs and captions, yielding the TikTok-SL-8 dataset (49 hours across eight languages). The authors benchmark two gloss-free SLT baselines, SEM-SLT and Signformer, showing SEM-SLT is more robust to slight data noisiness and benefits from pretraining when trained on automatically curated data. The approach enables scalable, weakly supervised pretraining for SLT and demonstrates practical data acquisition from platforms like TikTok, with potential to broaden multilingual coverage and data availability in sign-language research.

Abstract

Most existing sign language translation (SLT) datasets are limited in scale, lack multilingual coverage, and are costly to curate due to their reliance on expert annotation and controlled recording setup. Recently, Vision Language Models (VLMs) have demonstrated strong capabilities as evaluators and real-time assistants. Despite these advancements, their potential remains untapped in the context of sign language dataset acquisition. To bridge this gap, we introduce the first automated annotation and filtering framework that utilizes VLMs to reduce reliance on manual effort while preserving data quality. Our method is applied to TikTok videos across eight sign languages and to the already curated YouTube-SL-25 dataset in German Sign Language for the purpose of additional evaluation. Our VLM-based pipeline includes a face visibility detection, a sign activity recognition, a text extraction from video content, and a judgment step to validate alignment between video and text, implementing generic filtering, annotation and validation steps. Using the resulting corpus, TikTok-SL-8, we assess the performance of two off-the-shelf SLT models on our filtered dataset for German and American Sign Languages, with the goal of establishing baselines and evaluating the robustness of recent models on automatically extracted, slightly noisy data. Our work enables scalable, weakly supervised pretraining for SLT and facilitates data acquisition from social media.

Figures (7)

• Figure 1: An overview of our VLM-based SLT dataset collection framework on social media, with a particular focus on TikTok. The pipeline consists of three key stages: data filtering, data annotation, and data verification.
• Figure 2: Prompt template used for the VLM FaceDetector.
• Figure 3: Confusion matrices showing the agreement between our framework's automated labels and manual annotations for DGS and ASL subsets of TikTok-SL-8.
• Figure 4: Hashtags used for crawling videos for each sign language in the TikTok-SL-8 dataset.
• Figure 5: Prompt template used for the VLM SignActivityDetector.