Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

Roshan Sharma, Suwon Shon, Mark Lindsey, Hira Dhamyal, Rita Singh, Bhiksha Raj

TL;DR

The paper addresses whether human summaries of speech differ when based on listening to audio versus reading transcripts, introducing a large, expert-annotated dataset from the Interview corpus. It employs source-based, structure-based, and cross-summaries evaluations, including LLM and human AB tests, to show that speech-based summaries are more information-selective and factually consistent, while transcript-based summaries are more informative but vulnerable to ASR errors. The findings highlight the trade-offs between modality, transcription quality, and expertise, with expert annotations delivering more reliable and fluent outputs. By releasing the dataset and code, the work provides a foundation for better evaluation and development of speech summarization systems and cross-modal annotation practices.

Abstract

Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method. We find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Meanwhile, transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public(https://github.com/cmu-mlsp/interview_humanssum) to facilitate the reproduction of our work and advance research in this area.

Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

TL;DR

The paper addresses whether human summaries of speech differ when based on listening to audio versus reading transcripts, introducing a large, expert-annotated dataset from the Interview corpus. It employs source-based, structure-based, and cross-summaries evaluations, including LLM and human AB tests, to show that speech-based summaries are more information-selective and factually consistent, while transcript-based summaries are more informative but vulnerable to ASR errors. The findings highlight the trade-offs between modality, transcription quality, and expertise, with expert annotations delivering more reliable and fluent outputs. By releasing the dataset and code, the work provides a foundation for better evaluation and development of speech summarization systems and cross-modal annotation practices.

Abstract

Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method. We find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Meanwhile, transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public(https://github.com/cmu-mlsp/interview_humanssum) to facilitate the reproduction of our work and advance research in this area.
Paper Structure (32 sections, 10 figures, 4 tables)

This paper contains 32 sections, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Collection
  4. Evaluating Differences across Summaries
  5. Source-based Evaluation
  6. Structure Evaluation
  7. Summary Comparisons
  8. LLM-based Evaluation
  9. Human Evaluation
  10. Research Methodology
  11. RQ1: Speech versus Text Inputs
  12. RQ2: Impact of Transcript Errors on Human Summarization
  13. RQ3: Expert versus Non-Expert Annotation
  14. Experimental Results and Discussion
  15. RQ1: Speech versus Transcript Inputs
  16. ...and 17 more sections

Figures (10)

  • Figure 1: Overview of collection process and research questions
  • Figure 2: Pairwise Score Computation
  • Figure 3: Template for questions presented to ChatGPT. Similar questions are used to solicit human scores.
  • Figure 4: Scatterplot showing variance of entity F1 of summary compared to source reference transcript with Word Error Rate (WER) of audio transcription
  • Figure 5: Annotation guideline
  • ...and 5 more figures