Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AG-LSEC: Audio Grounded Lexical Speaker Error Correction

Rohit Paturi, Xiang Li, Sundararajan Srinivasan

TL;DR

This paper proposes to enhance and acoustically ground the LSEC system with speaker scores directly derived from the existing SD pipeline that achieves significant relative WDER reductions in the range of 25-40% over the audio-based SD, ASR system and beats the LSEC system by 15-25% relative on RT03-CTS, Callhome American English and Fisher datasets.

Abstract

Speaker Diarization (SD) systems are typically audio-based and operate independently of the ASR system in traditional speech transcription pipelines and can have speaker errors due to SD and/or ASR reconciliation, especially around speaker turns and regions of speech overlap. To reduce these errors, a Lexical Speaker Error Correction (LSEC), in which an external language model provides lexical information to correct the speaker errors, was recently proposed. Though the approach achieves good Word Diarization error rate (WDER) improvements, it does not use any additional acoustic information and is prone to miscorrections. In this paper, we propose to enhance and acoustically ground the LSEC system with speaker scores directly derived from the existing SD pipeline. This approach achieves significant relative WDER reductions in the range of 25-40% over the audio-based SD, ASR system and beats the LSEC system by 15-25% relative on RT03-CTS, Callhome American English and Fisher datasets.

AG-LSEC: Audio Grounded Lexical Speaker Error Correction

TL;DR

This paper proposes to enhance and acoustically ground the LSEC system with speaker scores directly derived from the existing SD pipeline that achieves significant relative WDER reductions in the range of 25-40% over the audio-based SD, ASR system and beats the LSEC system by 15-25% relative on RT03-CTS, Callhome American English and Fisher datasets.

Abstract

Speaker Diarization (SD) systems are typically audio-based and operate independently of the ASR system in traditional speech transcription pipelines and can have speaker errors due to SD and/or ASR reconciliation, especially around speaker turns and regions of speech overlap. To reduce these errors, a Lexical Speaker Error Correction (LSEC), in which an external language model provides lexical information to correct the speaker errors, was recently proposed. Though the approach achieves good Word Diarization error rate (WDER) improvements, it does not use any additional acoustic information and is prone to miscorrections. In this paper, we propose to enhance and acoustically ground the LSEC system with speaker scores directly derived from the existing SD pipeline. This approach achieves significant relative WDER reductions in the range of 25-40% over the audio-based SD, ASR system and beats the LSEC system by 15-25% relative on RT03-CTS, Callhome American English and Fisher datasets.
Paper Structure (17 sections, 9 equations, 3 figures, 2 tables)

This paper contains 17 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio Grounded Lexical Error Corrector
  4. Acoustic Speaker Scores
  5. End-to-end Neural Diarization
  6. Posterior post-processing
  7. Acoustic Grounding
  8. Early Fusion
  9. Late Fusion
  10. Training Methodology
  11. Inference Setup
  12. Experiments
  13. Data and Metrics
  14. Baseline System
  15. AG-LSEC System
  16. ...and 2 more sections

Figures (3)

  • Figure 1: (a) AG-LSEC with Late Fusion of Speaker Scores, (b) Speaker Scores Extraction form the SD pipeline, (c) AG-LSEC with Early Fusion of Speaker Scores
  • Figure 2: Ablation of the AG-LSEC models with different amounts of training data and initializations.
  • Figure 3: Qualitative examples of over-correction and under-correction with LSEC model rectified by the AG-LSEC Early Fusion model. The words highlighted in red are speaker errors.