LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech
Fei Yang, Xuanfan Ni, Renyi Yang, Jiahui Geng, Qing Li, Chenyang Lyu, Yichao Du, Longyue Wang, Weihua Luo, Kaifu Zhang
TL;DR
LongSpeech addresses the gap between short-form benchmarks and real-world long-form audio tasks by introducing a large-scale, multi-task benchmark with over 100k segments (~10 minutes each) and eight tasks including ASR, translation, summarization, and temporal reasoning. It provides a reproducible data construction pipeline from diverse public sources and unified train/dev/test splits totaling 142,200/30,100/30,100 examples, enabling robust evaluation of long-form audio-language models. Baseline experiments reveal pronounced gaps and model specialization, with higher-level tasks such as temporal localization and summarization proving particularly challenging. By releasing this benchmark to the community, the work aims to catalyze the development of robust, discourse-aware models capable of maintaining context and reasoning over extended audio.
Abstract
Recent advances in audio-language models have demonstrated remarkable success on short, segment-level speech tasks. However, real-world applications such as meeting transcription, spoken document understanding, and conversational analysis require robust models capable of processing and reasoning over long-form audio. In this work, we present LongSpeech, a large-scale and scalable benchmark specifically designed to evaluate and advance the capabilities of speech models on long-duration audio. LongSpeech comprises over 100,000 speech segments, each approximately 10 minutes long, with rich annotations for ASR, speech translation, summarization, language detection, speaker counting, content separation, and question answering. We introduce a reproducible pipeline for constructing long-form speech benchmarks from diverse sources, enabling future extensions. Our initial experiments with state-of-the-art models reveal significant performance gaps, with models often specializing in one task at the expense of others and struggling with higher-level reasoning. These findings underscore the challenging nature of our benchmark. Our benchmark will be made publicly available to the research community.