Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks

Supriti Sinhamahapatra, Jan Niehues

TL;DR

The paper tackles the difficulty of transcribing domain-specific vocabulary in scientific talks by incorporating visual slide context into automatic transcription. It builds a multi-modal benchmark by extending ACL datasets with slide information and introduces domain-specific word metrics to quantify improvements, while evaluating baselines such as Whisper, SALMONN, and Phi-4-multimodal. The authors propose cascaded and end-to-end context integration, plus a data-augmentation pipeline that generates slide-like visuals via LLaMA-3 and uses OCR/vision-language tools to extract domain terms, achieving about a 34%–35% relative reduction in domain-word WER versus the audio-only baseline. The results demonstrate that including slide-derived context—textual or visual—significantly improves transcription of specialized terminology, with fine-tuning on augmented data yielding the strongest gains and suggesting practical benefits for conference transcription and scientific communication.

Abstract

State-of-the-art (SOTA) Automatic Speech Recognition (ASR) systems primarily rely on acoustic information while disregarding additional multi-modal context. However, visual information are essential in disambiguation and adaptation. While most work focus on speaker images to handle noise conditions, this work also focuses on integrating presentation slides for the use cases of scientific presentation. In a first step, we create a benchmark for multi-modal presentation including an automatic analysis of transcribing domain-specific terminology. Next, we explore methods for augmenting speech models with multi-modal information. We mitigate the lack of datasets with accompanying slides by a suitable approach of data augmentation. Finally, we train a model using the augmented dataset, resulting in a relative reduction in word error rate of approximately 34%, across all words and 35%, for domain-specific terms compared to the baseline model.

Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks

TL;DR

The paper tackles the difficulty of transcribing domain-specific vocabulary in scientific talks by incorporating visual slide context into automatic transcription. It builds a multi-modal benchmark by extending ACL datasets with slide information and introduces domain-specific word metrics to quantify improvements, while evaluating baselines such as Whisper, SALMONN, and Phi-4-multimodal. The authors propose cascaded and end-to-end context integration, plus a data-augmentation pipeline that generates slide-like visuals via LLaMA-3 and uses OCR/vision-language tools to extract domain terms, achieving about a 34%–35% relative reduction in domain-word WER versus the audio-only baseline. The results demonstrate that including slide-derived context—textual or visual—significantly improves transcription of specialized terminology, with fine-tuning on augmented data yielding the strongest gains and suggesting practical benefits for conference transcription and scientific communication.

Abstract

State-of-the-art (SOTA) Automatic Speech Recognition (ASR) systems primarily rely on acoustic information while disregarding additional multi-modal context. However, visual information are essential in disambiguation and adaptation. While most work focus on speaker images to handle noise conditions, this work also focuses on integrating presentation slides for the use cases of scientific presentation. In a first step, we create a benchmark for multi-modal presentation including an automatic analysis of transcribing domain-specific terminology. Next, we explore methods for augmenting speech models with multi-modal information. We mitigate the lack of datasets with accompanying slides by a suitable approach of data augmentation. Finally, we train a model using the augmented dataset, resulting in a relative reduction in word error rate of approximately 34%, across all words and 35%, for domain-specific terms compared to the baseline model.
Paper Structure (31 sections, 3 figures, 5 tables)

This paper contains 31 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modal Scientific Presentation Benchmark
  4. Benchmark
  5. Metric
  6. Baseline
  7. Whisper:
  8. SALMONN:
  9. Phi-4-multimodal:
  10. Analysis
  11. Multi-modal Context Extraction and Integration
  12. Image Frame Extraction
  13. Text Extraction
  14. Context Integration
  15. Data Augmentation
  16. ...and 16 more sections

Figures (3)

  • Figure 1: An example of ASR transcription before and after using multi-modal input. Left: ASR baseline makes mistakes (in red) for multiple words. Right: Model correctly transcribes words (in blue) using multi-modal inputs.
  • Figure 2: Overview of our two approaches. The green arrows represent the end-to-end approach.
  • Figure 3: Example of transcriptions generated by different models with respect to the manual transcript. The figure shows that the best possible transcript is generated while fine-tuning the ASR model with llava prompts and image.