Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lend a Hand: Semi Training-Free Cued Speech Recognition via MLLM-Driven Hand Modeling for Barrier-free Communication

Guanjie Huang, Danny Hin Kwok Tsang, Li Liu

TL;DR

This work tackles automatic Cued Speech Recognition under data-scarce conditions by proposing STF-ACSR, a semi training-free framework that leverages Multimodal Large Language Models (MLLMs) for zero-shot hand shape/position recognition through the Chinese CS Prompt Module (CCSPM) and fuses it with lip-reading via the Minimalist Fusion Module (MFM). The CCSPM reduces hand recognition to keyframe classification using a hand keyframe filter, a 40-frame hand-positions/shapes support set, and a tailored prompting strategy, enabling training-free hand understanding. The method achieves state-of-the-art results on Mandarin CS benchmarks and the new MHI-MCCSD dataset, demonstrating robust performance for both normal and hearing-impaired cuers and highlighting the practical potential for barrier-free communication. The work also introduces a large, open Mandarin CS dataset with hearing-impaired participants, supporting broader evaluation and generalization to other language CS systems in future work.

Abstract

Cued Speech (CS) is an innovative visual communication system that integrates lip-reading with hand coding, designed to enhance effective communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) refers to the AI-driven process of automatically recognizing hand gestures and lip movements in CS, converting them into text. However, previous work often relies on complex fusion modules and training techniques. Additionally, due to the limited amount of data in CS, the extraction of hand features, as well as recognition modeling, has consistently been subpar, significantly limiting the effectiveness of ACSR. To address this issue, we have innovatively explored the capabilities of Multimodal large language models (MLLMs) in recognizing hand shapes and positions in CS. More precisely, we propose a new Semi Training-Free paradigm for ACSR, named STF-ACSR. This approach leverages zero-shot recognition of hand movements through the Chinese CS Prompt Module (CCSPM), which equipped a training-free keyframe filtering and customized prompt engineering based on MLLM. It then integrates the recognition results into the lip-reading model using a Minimalist Fusion Module (MFM), effectively achieving superior recognition results. Furthermore, specifically for this study, we have supplemented the existing dataset of 6 normal hearing CS cuers by recording additional data from 8 cuers with hearing impairments, resulting in a new mixed dataset. Extensive experiments have demonstrated that STF-ACSR significantly outperforms previous methods on both normal and hearing-impaired data. Implementation and checkpoints are available at https://github.com/DennisHgj/STF_ACSR.

Lend a Hand: Semi Training-Free Cued Speech Recognition via MLLM-Driven Hand Modeling for Barrier-free Communication

TL;DR

This work tackles automatic Cued Speech Recognition under data-scarce conditions by proposing STF-ACSR, a semi training-free framework that leverages Multimodal Large Language Models (MLLMs) for zero-shot hand shape/position recognition through the Chinese CS Prompt Module (CCSPM) and fuses it with lip-reading via the Minimalist Fusion Module (MFM). The CCSPM reduces hand recognition to keyframe classification using a hand keyframe filter, a 40-frame hand-positions/shapes support set, and a tailored prompting strategy, enabling training-free hand understanding. The method achieves state-of-the-art results on Mandarin CS benchmarks and the new MHI-MCCSD dataset, demonstrating robust performance for both normal and hearing-impaired cuers and highlighting the practical potential for barrier-free communication. The work also introduces a large, open Mandarin CS dataset with hearing-impaired participants, supporting broader evaluation and generalization to other language CS systems in future work.

Abstract

Cued Speech (CS) is an innovative visual communication system that integrates lip-reading with hand coding, designed to enhance effective communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) refers to the AI-driven process of automatically recognizing hand gestures and lip movements in CS, converting them into text. However, previous work often relies on complex fusion modules and training techniques. Additionally, due to the limited amount of data in CS, the extraction of hand features, as well as recognition modeling, has consistently been subpar, significantly limiting the effectiveness of ACSR. To address this issue, we have innovatively explored the capabilities of Multimodal large language models (MLLMs) in recognizing hand shapes and positions in CS. More precisely, we propose a new Semi Training-Free paradigm for ACSR, named STF-ACSR. This approach leverages zero-shot recognition of hand movements through the Chinese CS Prompt Module (CCSPM), which equipped a training-free keyframe filtering and customized prompt engineering based on MLLM. It then integrates the recognition results into the lip-reading model using a Minimalist Fusion Module (MFM), effectively achieving superior recognition results. Furthermore, specifically for this study, we have supplemented the existing dataset of 6 normal hearing CS cuers by recording additional data from 8 cuers with hearing impairments, resulting in a new mixed dataset. Extensive experiments have demonstrated that STF-ACSR significantly outperforms previous methods on both normal and hearing-impaired data. Implementation and checkpoints are available at https://github.com/DennisHgj/STF_ACSR.

Paper Structure

This paper contains 15 sections, 3 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Automatic Cued Speech Recognition
  4. Lip Reading Models
  5. Methods
  6. Problem Formulation
  7. Overview of STF-ACSR
  8. Chinese CS Prompt Module
  9. Minimalist Fusion Module
  10. New Mandarin Chinese CS Dataset
  11. Experiments
  12. Experiment Setups
  13. Comparative Study
  14. Ablation Study
  15. Conclusion

Figures (3)

  • Figure 1: Mandarin Chinese CS system. Five hand positions (mouth, chin, throat, side, cheek) and eight hand shapes are defined to encode Chinese vowels and consonants to assist lip reading (image from liu2019pilot).
  • Figure 2: Overview of STF-ACSR. It extracts hand information with the help of CCSPM and fuses it with lip features through MFM. CCSPM consists of a Hand Keyframe Filter, a positions and shapes support set, and a customized prompt template. MFM embeds the recognition results of MLLM and adjusts the dimension through a linear layer to achieve the fusion of hand and lip information. "CCSPM", "MFM", and "MLLM" are shorts for the Chinese Cued Speech Prompt Module, Minimalist Fusion Module, and Multimodal Large Language Model. For the hand embedding matrix, $T$ indicates the frame number, 44 is composed of 40 phonemes and four symbols.
  • Figure 3: Phoneme confusion matrices of STF-ACSR model on the test sets of multi-normal-cuer setting of MCCSD liu2023cross (six-hearing cuers, shows in subfigure. (a)) and MHI-MCCSD (eight hearing-impaired cuers, shows in subfigure. (b)). Results reveal phonemes that are easily confused.