SpeechEE: A Novel Benchmark for Speech Event Extraction

TL;DR

SpeechEE introduces a novel task of extracting structured event information directly from speech, addressing the gap where EE has been predominantly text-centric. The authors build a large, multi-domain benchmark by converting textual EE datasets into speech via manual narration and TTS synthesis in English and Chinese, then propose an end-to-end SpeechEE model that uses a Shrinking Unit, contrastive encoder learning, and a retrieval-aided decoder with an external Entity Dictionary. Experiments show the end-to-end approach consistently outperforms pipeline baselines across sentence, document, and dialogue data, with ablations demonstrating the critical contributions of CL, SU, and ED. The work establishes a strong baseline for SpeechEE, analyzes the challenges of speech-based EE, and outlines practical directions for future research, including noise robustness, cross-lingual transfer, and improved evaluation metrics.

Abstract

Event extraction (EE) is a critical direction in the field of information extraction, laying an important foundation for the construction of structured knowledge bases. EE from text has received ample research and attention for years, yet there can be numerous real-world applications that require direct information acquisition from speech signals, online meeting minutes, interview summaries, press releases, etc. While EE from speech has remained under-explored, this paper fills the gap by pioneering a SpeechEE, defined as detecting the event predicates and arguments from a given audio speech. To benchmark the SpeechEE task, we first construct a large-scale high-quality dataset. Based on textual EE datasets under the sentence, document, and dialogue scenarios, we convert texts into speeches through both manual real-person narration and automatic synthesis, empowering the data with diverse scenarios, languages, domains, ambiences, and speaker styles. Further, to effectively address the key challenges in the task, we tailor an E2E SpeechEE system based on the encoder-decoder architecture, where a novel Shrinking Unit module and a retrieval-aided decoding mechanism are devised. Extensive experimental results on all SpeechEE subsets demonstrate the efficacy of the proposed model, offering a strong baseline for the task. At last, being the first work on this topic, we shed light on key directions for future research.

Figures (10)

• Figure 1: An illustration of the speech event extraction's broad applications and task formulation.
• Figure 2: The architecture of the pipeline and E2E SpeechEE model.
• Figure 3: Comparisons between TextEE and SpeechEE.
• Figure 4: Analysis of the effect of synthesis dataset.
• Figure 5: T-SNE visualization about the effect of CL.