Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech

Deepak Babu Piskala

TL;DR

ProfASR-Bench introduces a context-conditioned ASR benchmark tailored for high-stakes professional domains (finance, medicine, legal, technology). It pairs natural-language prompts with entity-dense utterances to evaluate how context influences transcription across encoder–decoder ASR and audio-language models, using a standardized ladder of context (none, profile, domain+profile, oracle, adversarial) and entity-aware metrics. Across Whisper and Qwen baselines, results reveal a context-utilization gap where lightweight prompts barely affect average WER, though SER and entity-centric analyses show domain- and model-dependent shifts; adversarial prompts rarely degrade performance. The work provides a reproducible, multi-domain suite with demographic slices and robust reporting to accelerate development of on-the-fly, context-aware fusion strategies for real-world, high-stakes transcription tasks.

Abstract

Automatic Speech Recognition (ASR) in professional settings faces challenges that existing benchmarks underplay: dense domain terminology, formal register variation, and near-zero tolerance for critical entity errors. We present ProfASR-Bench, a professional-talk evaluation suite for high-stakes applications across finance, medicine, legal, and technology. Each example pairs a natural-language prompt (domain cue and/or speaker profile) with an entity-rich target utterance, enabling controlled measurement of context-conditioned recognition. The corpus supports conventional ASR metrics alongside entity-aware scores and slice-wise reporting by accent and gender. Using representative families Whisper (encoder-decoder ASR) and Qwen-Omni (audio language models) under matched no-context, profile, domain+profile, oracle, and adversarial conditions, we find a consistent pattern: lightweight textual context produces little to no change in average word error rate (WER), even with oracle prompts, and adversarial prompts do not reliably degrade performance. We term this the context-utilization gap (CUG): current systems are nominally promptable yet underuse readily available side information. ProfASR-Bench provides a standardized context ladder, entity- and slice-aware reporting with confidence intervals, and a reproducible testbed for comparing fusion strategies across model families. Dataset: https://huggingface.co/datasets/prdeepakbabu/ProfASR-Bench Code: https://github.com/prdeepakbabu/ProfASR-Bench

PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech

TL;DR

ProfASR-Bench introduces a context-conditioned ASR benchmark tailored for high-stakes professional domains (finance, medicine, legal, technology). It pairs natural-language prompts with entity-dense utterances to evaluate how context influences transcription across encoder–decoder ASR and audio-language models, using a standardized ladder of context (none, profile, domain+profile, oracle, adversarial) and entity-aware metrics. Across Whisper and Qwen baselines, results reveal a context-utilization gap where lightweight prompts barely affect average WER, though SER and entity-centric analyses show domain- and model-dependent shifts; adversarial prompts rarely degrade performance. The work provides a reproducible, multi-domain suite with demographic slices and robust reporting to accelerate development of on-the-fly, context-aware fusion strategies for real-world, high-stakes transcription tasks.

Abstract

Automatic Speech Recognition (ASR) in professional settings faces challenges that existing benchmarks underplay: dense domain terminology, formal register variation, and near-zero tolerance for critical entity errors. We present ProfASR-Bench, a professional-talk evaluation suite for high-stakes applications across finance, medicine, legal, and technology. Each example pairs a natural-language prompt (domain cue and/or speaker profile) with an entity-rich target utterance, enabling controlled measurement of context-conditioned recognition. The corpus supports conventional ASR metrics alongside entity-aware scores and slice-wise reporting by accent and gender. Using representative families Whisper (encoder-decoder ASR) and Qwen-Omni (audio language models) under matched no-context, profile, domain+profile, oracle, and adversarial conditions, we find a consistent pattern: lightweight textual context produces little to no change in average word error rate (WER), even with oracle prompts, and adversarial prompts do not reliably degrade performance. We term this the context-utilization gap (CUG): current systems are nominally promptable yet underuse readily available side information. ProfASR-Bench provides a standardized context ladder, entity- and slice-aware reporting with confidence intervals, and a reproducible testbed for comparing fusion strategies across model families. Dataset: https://huggingface.co/datasets/prdeepakbabu/ProfASR-Bench Code: https://github.com/prdeepakbabu/ProfASR-Bench
Paper Structure (15 sections, 3 equations, 3 figures, 6 tables)

This paper contains 15 sections, 3 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Context-conditioned ASR as a learning problem.
  3. Related Work
  4. Dataset: ProfASR-Bench
  5. Composition
  6. Entity types and distributions.
  7. Generation
  8. Purpose
  9. Results
  10. Evaluation pipeline.
  11. Metrics.
  12. Impact of Context
  13. Prompt conditions (with guiding examples).
  14. Conclusion
  15. Reproducibility Statement

Figures (3)

  • Figure 1: ProfASR at a glance. Four domain vignettes Medicine, Finance, Legal, and Technology illustrate prompt-conditioned ASR on professional talk. Each scene pairs the utterance with a previous-sentence prompt and a speaker profile, and highlights typed entities. Red marks indicate representative no-context errors on critical tokens (e.g., DRUG, MONEY/NUMERIC, MODALITY, VERSION). The figure motivates our evaluation: matched with/without-context comparisons centered on entity-aware metrics and slice-wise reporting, rather than average WER alone.
  • Figure 2: High-stakes ASR error: hydralazine → hydroxyzine.
  • Figure 3: Top-5 entity types by domain. Each bar reports within-domain percentage for the five most frequent entity types in Finance, Medicine, Legal, and Technology. The concentration of domain-critical categories motivates entity-centric evaluation alongside conventional WER.