Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus

Yexing Du, Kaiyuan Liu, Bihe Zhang, Youcheng Pan, Bo Yang, Liangyu Huo, Xiyuan Zhang, Jian Xie, Daojing He, Yang Xiang, Ming Liu, Bin Qin

TL;DR

MCGA introduces the Multi-task Classical Chinese Literary Genre Audio Corpus, a first-of-its-kind large-scale, copyright-cleared CCS audio dataset with 22,000 samples (119 hours) spanning five genres and 11 historical periods. It supports six speech tasks (ASR, S2TT, SEC, SQA, SU, SR) and four text tasks, enabling robust evaluation of Multimodal LLMs on domain-specific audio data. The authors propose a dedicated SEC evaluation metric and a Cross-Modal Consistency (CMC) measure, and they benchmark 10 MLLMs (2 closed-source, 8 open-source), revealing substantial room for improvement, especially in SEC and open-ended SQA. The work demonstrates substantial benefits from fine-tuning and provides public release of MCGA and code to foster further development of audio-enabled CCS models and tools, with clear implications for cultural preservation and scholarly inquiry.

Abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), their potential has garnered significant attention in Chinese Classical Studies (CCS). While existing research has primarily focused on text and visual modalities, the audio corpus within this domain remains largely underexplored. To bridge this gap, we propose the Multi-task Classical Chinese Literary Genre Audio Corpus (MCGA). It encompasses a diverse range of literary genres across six tasks: Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech Emotion Captioning (SEC), Spoken Question Answering (SQA), Speech Understanding (SU), and Speech Reasoning (SR). Through the evaluation of ten MLLMs, our experimental results demonstrate that current models still face substantial challenges when processed on the MCGA test set. Furthermore, we introduce an evaluation metric for SEC and a metric to measure the consistency between the speech and text capabilities of MLLMs. We release MCGA and our code to the public to facilitate the development of MLLMs with more robust multidimensional audio capabilities in CCS. MCGA Corpus: https://github.com/yxduir/MCGA

MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus

TL;DR

MCGA introduces the Multi-task Classical Chinese Literary Genre Audio Corpus, a first-of-its-kind large-scale, copyright-cleared CCS audio dataset with 22,000 samples (119 hours) spanning five genres and 11 historical periods. It supports six speech tasks (ASR, S2TT, SEC, SQA, SU, SR) and four text tasks, enabling robust evaluation of Multimodal LLMs on domain-specific audio data. The authors propose a dedicated SEC evaluation metric and a Cross-Modal Consistency (CMC) measure, and they benchmark 10 MLLMs (2 closed-source, 8 open-source), revealing substantial room for improvement, especially in SEC and open-ended SQA. The work demonstrates substantial benefits from fine-tuning and provides public release of MCGA and code to foster further development of audio-enabled CCS models and tools, with clear implications for cultural preservation and scholarly inquiry.

Abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), their potential has garnered significant attention in Chinese Classical Studies (CCS). While existing research has primarily focused on text and visual modalities, the audio corpus within this domain remains largely underexplored. To bridge this gap, we propose the Multi-task Classical Chinese Literary Genre Audio Corpus (MCGA). It encompasses a diverse range of literary genres across six tasks: Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech Emotion Captioning (SEC), Spoken Question Answering (SQA), Speech Understanding (SU), and Speech Reasoning (SR). Through the evaluation of ten MLLMs, our experimental results demonstrate that current models still face substantial challenges when processed on the MCGA test set. Furthermore, we introduce an evaluation metric for SEC and a metric to measure the consistency between the speech and text capabilities of MLLMs. We release MCGA and our code to the public to facilitate the development of MLLMs with more robust multidimensional audio capabilities in CCS. MCGA Corpus: https://github.com/yxduir/MCGA
Paper Structure (43 sections, 1 equation, 7 figures, 8 tables)

This paper contains 43 sections, 1 equation, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Chinese Cultural Datasets
  4. Chinese Classical Studies Datasets
  5. MCGA Corpus
  6. Overview
  7. Data Construction
  8. Data Collection and Preprocessing.
  9. Text Data Construction.
  10. Text Data Verification.
  11. Human Recording
  12. Volunteer Demographics.
  13. Recording Protocol.
  14. Audio Quality Check
  15. MLLM Verification.
  16. ...and 28 more sections

Figures (7)

  • Figure 1: Timeline of the Golden Age for Classical Chinese Literary Genres: Fu (Rhapsody), Shi (Poetry), Wen (Prose), Ci (Lyric), and Qu (Song).
  • Figure 2: Examples from the MCGA Corpus. The corpus covers six core speech tasks (ASR, S2TT, SEC, SQA, SU, SR). Leveraging its parallel speech-text data, it also supports four text tasks: Machine Translation (MT), Question Answering (QA), Language Understanding (LU), and Language Reasoning (LR).
  • Figure 3: MCGA Corpus Construction. Initially comprising only metadata such as titles, authors, and texts, the MCGA corpus is expanded through human recording, LLM generation, and rigorous verification. Then, it supports six speech tasks: ASR, S2T, SEC, SQA, SU, and SR. We provide a detailed example of the SEC task in Figure \ref{['case_sec']}.
  • Figure 4: Case for SEC Task.
  • Figure 5: Corpus Statistics. It comprises 22,000 filtered human-recorded speech samples (totaling 119 hours) and supports 6 downstream tasks. Sample counts for S2TT, SEC, SU, and SR are lower due to the removal of invalid QA pairs. (NSD: the Northern and Southern Dynasties; FD: the Five Dynasties)
  • ...and 2 more figures