Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?

Yiwen Guan, Viet Anh Trinh, Vivek Voleti, Jacob Whitehill

TL;DR

The paper investigates when multiple input modalities improve decoder-only discrete-token ASR by extending a multimodal language model to fuse audio, image context, lip movements, and OCR. It introduces a controllable synthetic dataset (3‑Equations) and validates on SlideAVSR, showing that combining modalities generally boosts accuracy, with OCR providing the most consistent single-modality gains and image information offering benefits at moderate noise levels. The study further demonstrates that filtering irrelevant visual content before fusion enhances performance, and that modality benefits can vary with synchronization and noise, highlighting practical guidelines for multimodal ASR design. These findings advance understanding of multimodal fusion in ASR and have implications for educational AI and video conferencing applications, while acknowledging limitations of synthetic data and the value of exploring additional backbones and real-world datasets.

Abstract

Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities impact performance in specific scenarios remain limited. In this paper, we investigate the effects of multiple modalities on recognition accuracy on both synthetic and real-world datasets. Our experiments suggest that: (1) Integrating more modalities can increase accuracy; in particular, our paper is, to our best knowledge, the first to show the benefit of combining audio, image context, and lip information; (2) Images as a supplementary modality for speech recognition provide the greatest benefit at moderate noise levels, moreover, they exhibit a different trend compared to inherently synchronized modalities like lip movements; (3) Performance improves on both synthetic and real-world datasets when the most relevant visual information is filtered as a preprocessing step.

Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?

TL;DR

The paper investigates when multiple input modalities improve decoder-only discrete-token ASR by extending a multimodal language model to fuse audio, image context, lip movements, and OCR. It introduces a controllable synthetic dataset (3‑Equations) and validates on SlideAVSR, showing that combining modalities generally boosts accuracy, with OCR providing the most consistent single-modality gains and image information offering benefits at moderate noise levels. The study further demonstrates that filtering irrelevant visual content before fusion enhances performance, and that modality benefits can vary with synchronization and noise, highlighting practical guidelines for multimodal ASR design. These findings advance understanding of multimodal fusion in ASR and have implications for educational AI and video conferencing applications, while acknowledging limitations of synthetic data and the value of exploring additional backbones and real-world datasets.

Abstract

Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities impact performance in specific scenarios remain limited. In this paper, we investigate the effects of multiple modalities on recognition accuracy on both synthetic and real-world datasets. Our experiments suggest that: (1) Integrating more modalities can increase accuracy; in particular, our paper is, to our best knowledge, the first to show the benefit of combining audio, image context, and lip information; (2) Images as a supplementary modality for speech recognition provide the greatest benefit at moderate noise levels, moreover, they exhibit a different trend compared to inherently synchronized modalities like lip movements; (3) Performance improves on both synthetic and real-world datasets when the most relevant visual information is filtered as a preprocessing step.
Paper Structure (14 sections, 6 figures, 2 tables)

This paper contains 14 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets
  4. 3-Equations
  5. SlideAVSR
  6. Experimental Setup
  7. Model
  8. Model Configurations and Training Details
  9. Evaluation Metrics
  10. Results and Discussion
  11. Do additional modalities always help ASR accuracy?
  12. Does each modality provide a uniform accuracy boost?
  13. How does irrelevant visual info. affect ASR performance?
  14. Conclusions

Figures (6)

  • Figure 1: An example of the 3-Equations dataset. From left to right: the image sample shows 3 mathematical equations; the OCR texts are extracted with EasyOCR and cleaned to retain only numbers, letters, and operators; the audio contains randomly reading 2 out of the 3 equations aloud; the lip movement video displays a lip region reading the corresponding equation sentences; the ground-truth transcription is the plain-text translation (label) of the speech. In this example, the speech reads the third and the second equations in order.
  • Figure 2: Examples of Gemini's output on the 3-Equations 2-noise. For a fair comparison, we "help" Gemini spell out some words it recognizes as symbols. REF stands for reference, HYP stands for model's hypothesis.
  • Figure 3: An overview of Discrete Multimodal Language Model (DMLM). The inputs are encoded by modality-specific encoders, and the encoded multi-modal tokens are concatenated to a task prompt, and then passed to DMLM for processing in next-token prediction fashion.
  • Figure 4: Relative WER benefit (%) of adding single image, OCR, or lip modalities on 3-Equations 2-noise test set.
  • Figure 5: Relative WER benefit (%) of adding OCR with different K values for FQ Ranker on SlideAVSR test set.
  • ...and 1 more figures