Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VALLR-Pin: Dual-Decoding Visual Speech Recognition for Mandarin with Pinyin-Guided LLM Refinement

Chang Sun, Dongliang Xie, Wanpeng Xie, Bo Qin, Hong Yang

TL;DR

Mandarin visual speech recognition suffers from strong viseme ambiguity and high homophone prevalence. The authors propose VALLR-Pin, a two-stage framework that first uses a dual-decoder VSR to predict both Mandarin characters and Pinyin, then refines the outputs with an LLM guided by Pinyin and a set of N-best hypotheses, with the LLM further adapted via LoRA on error-aware data. The LLM is trained on model-generated error patterns to better map imperfect Pinyin inputs and competing hypotheses to accurate characters. Experiments on CNVSRC benchmarks and a self-collected dataset show consistent CER improvements, with ablations confirming the benefits of the dual-decoder design and LLM fine-tuning, indicating strong practical gains for Mandarin lip-reading in diverse conditions.

Abstract

Visual Speech Recognition aims to transcribe spoken words from silent lip-motion videos. This task is particularly challenging for Mandarin, as visemes are highly ambiguous and homophones are prevalent. We propose VALLR-Pin, a novel two-stage framework that extends the recent VALLR architecture from English to Mandarin. First, a shared video encoder feeds into dual decoders, which jointly predict both Chinese character sequences and their standard Pinyin romanization. The multi-task learning of character and phonetic outputs fosters robust visual-semantic representations. During inference, the text decoder generates multiple candidate transcripts. We construct a prompt by concatenating the Pinyin output with these candidate Chinese sequences and feed it to a large language model to resolve ambiguities and refine the transcription. This provides the LLM with explicit phonetic context to correct homophone-induced errors. Finally, we fine-tune the LLM on synthetic noisy examples: we generate imperfect Pinyin-text pairs from intermediate VALLR-Pin checkpoints using the training data, creating instruction-response pairs for error correction. This endows the LLM with awareness of our model's specific error patterns. In summary, VALLR-Pin synergizes visual features with phonetic and linguistic context to improve Mandarin lip-reading performance.

VALLR-Pin: Dual-Decoding Visual Speech Recognition for Mandarin with Pinyin-Guided LLM Refinement

TL;DR

Mandarin visual speech recognition suffers from strong viseme ambiguity and high homophone prevalence. The authors propose VALLR-Pin, a two-stage framework that first uses a dual-decoder VSR to predict both Mandarin characters and Pinyin, then refines the outputs with an LLM guided by Pinyin and a set of N-best hypotheses, with the LLM further adapted via LoRA on error-aware data. The LLM is trained on model-generated error patterns to better map imperfect Pinyin inputs and competing hypotheses to accurate characters. Experiments on CNVSRC benchmarks and a self-collected dataset show consistent CER improvements, with ablations confirming the benefits of the dual-decoder design and LLM fine-tuning, indicating strong practical gains for Mandarin lip-reading in diverse conditions.

Abstract

Visual Speech Recognition aims to transcribe spoken words from silent lip-motion videos. This task is particularly challenging for Mandarin, as visemes are highly ambiguous and homophones are prevalent. We propose VALLR-Pin, a novel two-stage framework that extends the recent VALLR architecture from English to Mandarin. First, a shared video encoder feeds into dual decoders, which jointly predict both Chinese character sequences and their standard Pinyin romanization. The multi-task learning of character and phonetic outputs fosters robust visual-semantic representations. During inference, the text decoder generates multiple candidate transcripts. We construct a prompt by concatenating the Pinyin output with these candidate Chinese sequences and feed it to a large language model to resolve ambiguities and refine the transcription. This provides the LLM with explicit phonetic context to correct homophone-induced errors. Finally, we fine-tune the LLM on synthetic noisy examples: we generate imperfect Pinyin-text pairs from intermediate VALLR-Pin checkpoints using the training data, creating instruction-response pairs for error correction. This endows the LLM with awareness of our model's specific error patterns. In summary, VALLR-Pin synergizes visual features with phonetic and linguistic context to improve Mandarin lip-reading performance.
Paper Structure (25 sections, 17 equations, 1 figure, 4 tables)

This paper contains 25 sections, 17 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Proposal
  3. Problem Formulation
  4. Stage I: Dual-Decoder Visual Speech Recognition
  5. Visual Frontend and Encoder
  6. Dual-Decoder Multi-Task Learning
  7. Stage II: LLM-Based Refinement with Pinyin Guidance
  8. N-best Hypothesis Generation
  9. LLM Fine-Tuning Dataset Construction
  10. LLM Fine-Tuning with LoRA
  11. Inference Summary
  12. Experiments
  13. Datasets
  14. CNVSRC-Multi.Dev.
  15. Self-Collected Dataset.
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Overview of the proposed VALLR-Pin framework. The system consists of a dual-decoder VSR model that predicts character and Pinyin sequences, followed by an LLM-based refinement module. Error-aware instruction data are constructed from model-generated hypotheses and used to adapt the LLM for Mandarin visual speech recognition.