Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, Guoqing Zhao

TL;DR

This paper tackles gloss-free sign language translation by identifying that end-to-end incorporation of large language models (LLMs) can hinder learning of visual sign representations. It introduces FLa-LLM, a factorized two-stage framework consisting of a Visual Initialing stage that pre-trains the visual encoder with a lightweight translator, and an LLM Fine-tuning stage that freezes the visual encoder while adapting a multilingual LLM (MBart) via an LLM-Adapter. The approach employs a ResNet-18 visual backbone with a local temporal module, a Visual-Language Adapter, and a Light-T translator, plus downsampling of videos to reduce computation; the LLM is fine-tuned with seq2seq cross-entropy losses, and inference uses beam search. Across PHOENIX14T, CSL-Daily, and How2Sign under gloss-free protocols, FLa-LLM achieves substantial BLEU-4 and ROUGE-L gains, surpassing prior gloss-free methods and approaching gloss-based performance, thus demonstrating practical potential for scalable SLT without gloss annotations.

Abstract

Previous Sign Language Translation (SLT) methods achieve superior performance by relying on gloss annotations. However, labeling high-quality glosses is a labor-intensive task, which limits the further development of SLT. Although some approaches work towards gloss-free SLT through jointly training the visual encoder and translation network, these efforts still suffer from poor performance and inefficient use of the powerful Large Language Model (LLM). Most seriously, we find that directly introducing LLM into SLT will lead to insufficient learning of visual representations as LLM dominates the learning curve. To address these problems, we propose Factorized Learning assisted with Large Language Model (FLa-LLM) for gloss-free SLT. Concretely, we factorize the training process into two stages. In the visual initialing stage, we employ a lightweight translation model after the visual encoder to pre-train the visual encoder. In the LLM fine-tuning stage, we freeze the acquired knowledge in the visual encoder and integrate it with a pre-trained LLM to inspire the LLM's translation potential. This factorized training strategy proves to be highly effective as evidenced by significant improvements achieved across three SLT datasets which are all conducted under the gloss-free setting.

Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

TL;DR

This paper tackles gloss-free sign language translation by identifying that end-to-end incorporation of large language models (LLMs) can hinder learning of visual sign representations. It introduces FLa-LLM, a factorized two-stage framework consisting of a Visual Initialing stage that pre-trains the visual encoder with a lightweight translator, and an LLM Fine-tuning stage that freezes the visual encoder while adapting a multilingual LLM (MBart) via an LLM-Adapter. The approach employs a ResNet-18 visual backbone with a local temporal module, a Visual-Language Adapter, and a Light-T translator, plus downsampling of videos to reduce computation; the LLM is fine-tuned with seq2seq cross-entropy losses, and inference uses beam search. Across PHOENIX14T, CSL-Daily, and How2Sign under gloss-free protocols, FLa-LLM achieves substantial BLEU-4 and ROUGE-L gains, surpassing prior gloss-free methods and approaching gloss-based performance, thus demonstrating practical potential for scalable SLT without gloss annotations.

Abstract

Previous Sign Language Translation (SLT) methods achieve superior performance by relying on gloss annotations. However, labeling high-quality glosses is a labor-intensive task, which limits the further development of SLT. Although some approaches work towards gloss-free SLT through jointly training the visual encoder and translation network, these efforts still suffer from poor performance and inefficient use of the powerful Large Language Model (LLM). Most seriously, we find that directly introducing LLM into SLT will lead to insufficient learning of visual representations as LLM dominates the learning curve. To address these problems, we propose Factorized Learning assisted with Large Language Model (FLa-LLM) for gloss-free SLT. Concretely, we factorize the training process into two stages. In the visual initialing stage, we employ a lightweight translation model after the visual encoder to pre-train the visual encoder. In the LLM fine-tuning stage, we freeze the acquired knowledge in the visual encoder and integrate it with a pre-trained LLM to inspire the LLM's translation potential. This factorized training strategy proves to be highly effective as evidenced by significant improvements achieved across three SLT datasets which are all conducted under the gloss-free setting.
Paper Structure (20 sections, 7 equations, 4 figures, 10 tables)

This paper contains 20 sections, 7 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Dominance of LLM in SLT
  4. Method
  5. Overview
  6. Visual Initialing
  7. LLM Fine-tuning
  8. Experiments
  9. Datasets and Evaluation Metrics
  10. Implementation Details
  11. Comparison with State-of-the-art Methods
  12. Ablation Studies
  13. Ablation on Factorized Learning
  14. Ablation on Visual Initializing
  15. Ablation on LLM Fine-tuning
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Different frameworks and performance of gloss-based and gloss-free methods with LLM. The first row shows the performance of the gloss-based method with LLM chen2022simple. The second and third show the gloss-free method with LLM in our experiments. The BLEU-4 score is gotten on the PHOENIX-2014T test set.
  • Figure 2: The grad norm and parameters norm of the last layer of the visual encoder and the last layer of the LLM when jointly training the visual encoder and LLM end-to-end.
  • Figure 3: The overall framework of our proposed method. LLM represents the large language model.
  • Figure 4: Effect of initialing time. VIS represents the visual initialing stage and LFS represents the LLM fine-tuning stage.