Improving Continuous Sign Language Recognition with Adapted Image Models

TL;DR

CSLR is hindered by data scarcity and the prohibitive cost of fine-tuning large vision-language models. AdaptSign tackles this by freezing a CLIP backbone and attaching lightweight adapters, prefix embeddings, multiscale feature aggregation, and cross-frame attention to model spatial and temporal sign cues, incurring only about $3.2\%$ extra computation. Across PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL, AdaptSign achieves state-of-the-art accuracy with strong ablations and insightful visualizations that show attention to hands, face, and motion trajectories. This approach enables efficient, generalizable CSLR with preserved pretrained knowledge, making deployment more feasible in data-scarce settings.

Abstract

The increase of web-scale weakly labelled image-text pairs have greatly facilitated the development of large-scale vision-language models (e.g., CLIP), which have shown impressive generalization performance over a series of downstream tasks. However, the massive model size and scarcity of available data limit their applications to fine-tune the whole model in downstream tasks. Besides, fully fine-tuning the model easily forgets the generic essential knowledge acquired in the pretraining stage and overfits the downstream data. To enable high efficiency when adapting these large vision-language models (e.g., CLIP) to performing continuous sign language recognition (CSLR) while preserving their generalizability, we propose a novel strategy (AdaptSign). Especially, CLIP is adopted as the visual backbone to extract frame-wise features whose parameters are fixed, and a set of learnable modules are introduced to model spatial sign variations or capture temporal sign movements. The introduced additional modules are quite lightweight, only owning 3.2% extra computations with high efficiency. The generic knowledge acquired in the pretraining stage is well-preserved in the frozen CLIP backbone in this process. Extensive experiments show that despite being efficient, AdaptSign is able to demonstrate superior performance across a series of CSLR benchmarks including PHOENIX14, PHOENIX14-T, CSL-Daily and CSL compared to existing methods. Visualizations show that AdaptSign could learn to dynamically pay major attention to the informative spatial regions and cross-frame trajectories in sign videos.

Figures (4)

• Figure 1: Illustration of the difference between our training pipeline and the commonly-used pretrain-and-finetune paradigm.
• Figure 2: An overview for the framework of AdaptSign. We instantiate a frozen CLIP model as the backbone of spatial extractor, with several proposed modules on top for feature calibration. A sequence model is attached to perform sentence prediction following previous CSLR methods Min_2021_ICCVcheng2020fullyhao2021selfhu2022temporalhu2023self. The high-quality features of the frozen CLIP backbone are efficiently transferred through the proposed lightweight modules to perform CSLR.
• Figure 3: Visualizations of spatial attention maps. Top: raw frames; Middle: attention maps of a frozen CLIP backbone; Bottom: attention maps of ours. The red box denotes the query location. Compared to the frozen CLIP backbone, our method could learn to emphasize domain-sensitive regions like hands and face (dark red areas) which play an important role in expressing sign language.
• Figure 4: Visualizations of attention maps generated by our cross-frame attention module. The red box denotes the query location, i.e., the frame-level token $x_{cls}^{"}$. It's observed that the query could always attend to informative regions in neighboring frames, e.g., hands or face, to track critical body trajectories in expressing a sign.