WESR: Scaling and Evaluating Word-level Event-Speech Recognition

TL;DR

WESR advances speech understanding by jointly transcribing spoken content and word-aligned non-verbal vocal events. It defines a 21-category taxonomy split into discrete and continuous events, and introduces WESR-Bench, a 900+ utterance benchmark with a position-aware evaluation protocol that decouples ASR errors from event localization. A large training corpus, WESR-Train, enables specialized models that outperform strong open-source baselines and commercial APIs while preserving ASR quality. Together, WESR provides a data-centric framework and evaluation standard to study rich, real-world auditory scenes and supports future multilingual, paralinguistic research.

Abstract

Speech conveys not only linguistic information but also rich non-verbal vocal events such as laughing and crying. While semantic transcription is well-studied, the precise localization of non-verbal events remains a critical yet under-explored challenge. Current methods suffer from insufficient task definitions with limited category coverage and ambiguous temporal granularity. They also lack standardized evaluation frameworks, hindering the development of downstream applications. To bridge this gap, we first develop a refined taxonomy of 21 vocal events, with a new categorization into discrete (standalone) versus continuous (mixed with speech) types. Based on the refined taxonomy, we introduce WESR-Bench, an expert-annotated evaluation set (900+ utterances) with a novel position-aware protocol that disentangles ASR errors from event detection, enabling precise localization measurement for both discrete and continuous events. We also build a strong baseline by constructing a 1,700+ hour corpus, and train specialized models, surpassing both open-source audio-language models and commercial APIs while preserving ASR quality. We anticipate that WESR will serve as a foundational resource for future research in modeling rich, real-world auditory scenes.

Figures (6)

• Figure 1: Task overview of our WESR that generates transcripts with explicit word-aligned continuous/discrete event tags.
• Figure 2: Tag distribution in WESR-Bench. The inner ring shows major event categories, each represented by a distinct color. The outer ring displays the specific tag instances within each category.
• Figure 3: Illustration of Steps 2 and 3 of the WESR evaluation method after Event-Preserving Alignment. Event tags are extracted and mapped to their corresponding word or inter-word positions for metrics calculation.
• Figure 4: Performance comparison between continuous and discrete event tags on WESR-Bench.
• Figure 5: Tag distribution of WESR-Train.