Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OCR-Enhanced Multimodal ASR Can Read While Listening

Junli Chen, Changli Tang, Yixuan Li, Guangzhi Sun, Chao Zhang

TL;DR

Donut-Whisper presents an end-to-end audio-visual ASR framework that fuses Whisper-based audio with Donut-based OCR visual features through a dual-encoder and a cross-attention fusion pathway, enabling OCR-assisted recognition for English and Chinese. The approach introduces a sliding-window Q-Former to align audio with local visual context and demonstrates the value of a lightweight distillation scheme to transfer multimodal knowledge to an audio-only student. A multilingual AVSR dataset of movie clips with synchronized subtitles supports evaluation, showing significant gains over strong unimodal baselines. The work suggests practical impact in video transcription and multilingual AVSR, offering a roadmap for stronger multimodal fusion and iterative self-training with OCR cues.

Abstract

Visual information, such as subtitles in a movie, often helps automatic speech recognition. In this paper, we propose Donut-Whisper, an audio-visual ASR model with dual encoder to leverage visual information to improve speech recognition performance in both English and Chinese. Donut-Whisper combines the advantage of the linear and the Q-Former-based modality alignment structures via a cross-attention module, generating more powerful audio-visual features. Meanwhile, we propose a lightweight knowledge distillation scheme showcasing the potential of using audio-visual models to teach audio-only models to achieve better performance. Moreover, we propose a new multilingual audio-visual speech recognition dataset based on movie clips containing both Chinese and English partitions. As a result, Donut-Whisper achieved significantly better performance on both English and Chinese partition of the dataset compared to both Donut and Whisper large V3 baselines. In particular, an absolute 5.75% WER reduction and a 16.5% absolute CER reduction were achieved on the English and Chinese sets respectively compared to the Whisper ASR baseline.

OCR-Enhanced Multimodal ASR Can Read While Listening

TL;DR

Donut-Whisper presents an end-to-end audio-visual ASR framework that fuses Whisper-based audio with Donut-based OCR visual features through a dual-encoder and a cross-attention fusion pathway, enabling OCR-assisted recognition for English and Chinese. The approach introduces a sliding-window Q-Former to align audio with local visual context and demonstrates the value of a lightweight distillation scheme to transfer multimodal knowledge to an audio-only student. A multilingual AVSR dataset of movie clips with synchronized subtitles supports evaluation, showing significant gains over strong unimodal baselines. The work suggests practical impact in video transcription and multilingual AVSR, offering a roadmap for stronger multimodal fusion and iterative self-training with OCR cues.

Abstract

Visual information, such as subtitles in a movie, often helps automatic speech recognition. In this paper, we propose Donut-Whisper, an audio-visual ASR model with dual encoder to leverage visual information to improve speech recognition performance in both English and Chinese. Donut-Whisper combines the advantage of the linear and the Q-Former-based modality alignment structures via a cross-attention module, generating more powerful audio-visual features. Meanwhile, we propose a lightweight knowledge distillation scheme showcasing the potential of using audio-visual models to teach audio-only models to achieve better performance. Moreover, we propose a new multilingual audio-visual speech recognition dataset based on movie clips containing both Chinese and English partitions. As a result, Donut-Whisper achieved significantly better performance on both English and Chinese partition of the dataset compared to both Donut and Whisper large V3 baselines. In particular, an absolute 5.75% WER reduction and a 16.5% absolute CER reduction were achieved on the English and Chinese sets respectively compared to the Whisper ASR baseline.
Paper Structure (17 sections, 4 equations, 2 figures, 4 tables)

This paper contains 17 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio–Visual Models with Lip or Facial Cues
  4. Models Leveraging Visual Text
  5. Method
  6. Model Architecture
  7. Knowledge Distillation
  8. Experiments
  9. Dataset
  10. Baselines and Metrics
  11. Comparative Experiments
  12. Training Details
  13. Results and Analysis
  14. Main Results
  15. Effect of Distillation
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Architecture of the proposed Donut-Whisper model. Bottom left are the audio and visual encoders. The alignment module is shown in the dashed box. The decoder is on the right.
  • Figure 2: Architecture for alternative fusion structures under study in this paper, including a linear projection layer, a Q-Former and the proposed fusion module in Donut-Whisper.