Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation

Kohei Matsuura, Takanori Ashihara, Takafumi Moriya, Masato Mimura, Takatomo Kano, Atsunori Ogawa, Marc Delcroix

TL;DR

This work defines sentence-wise speech summarization (Sen-SSum) to enable real-time, sentence-by-sentence summaries of spoken content, bridging ASR and speech summarization. It introduces Mega-SSum (3.8M synthesized English triplets) and CSJ-SSum (38k real Japanese triplets) and compares cascade ASR+TSum with end-to-end models, finding end-to-end performance lags behind cascade due to data limits. A sequence-level knowledge distillation approach uses pseudo-summaries from a cascade model to train an end-to-end system, significantly boosting its performance on both datasets and even yielding pseudo-summaries that rival or exceed some human summaries in certain settings. The results highlight the practical viability of Sen-SSum for real-time, readable summaries and suggest promising directions, including integrating language models directly into E2E SSum and developing context-aware, long-document processing models.

Abstract

This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.

Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation

TL;DR

This work defines sentence-wise speech summarization (Sen-SSum) to enable real-time, sentence-by-sentence summaries of spoken content, bridging ASR and speech summarization. It introduces Mega-SSum (3.8M synthesized English triplets) and CSJ-SSum (38k real Japanese triplets) and compares cascade ASR+TSum with end-to-end models, finding end-to-end performance lags behind cascade due to data limits. A sequence-level knowledge distillation approach uses pseudo-summaries from a cascade model to train an end-to-end system, significantly boosting its performance on both datasets and even yielding pseudo-summaries that rival or exceed some human summaries in certain settings. The results highlight the practical viability of Sen-SSum for real-time, readable summaries and suggest promising directions, including integrating language models directly into E2E SSum and developing context-aware, long-document processing models.

Abstract

This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.
Paper Structure (16 sections, 2 equations, 4 figures, 3 tables)

This paper contains 16 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets for Sen-SSum
  4. Method
  5. Cascade and end-to-end modeling
  6. Sequence-level knowledge distillation for E2E models
  7. Leveraging self-supervised models
  8. Experiment
  9. Setup
  10. Model architectures
  11. Training details: datasets and hyper-parameters
  12. Evaluation details
  13. Results
  14. Results on Mega-SSum
  15. Results on CSJ-SSum
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Examples of ASR, Sen-SSum, and SSum. Sen-SSum combines real-time processing and conciseness.
  • Figure 2: ROUGE-L ($\uparrow$) and BERTScore ($\uparrow$) by cascade and E2E models trained with various numbers of human summaries "*-HS". "E2E-KD" is the results of E2E model trained with pseudo-summaries (i.e., knowledge distillation).
  • Figure 3: Results of the A/B test by ChatGPT. With more pseudo-summaries for training, more summaries by the E2E model were preferred to ones by the cascade baseline model.
  • Figure 4: "E2E-KD (ref)" shows the improvements by the knowledge distillation with reference transcriptions, which also improved the baseline models "*-base".