Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harmonizing the Arabic Audio Space with Data Scheduling

Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury

TL;DR

This work studies how data scheduling affects the adaptation of an Arabic-centric audio LLM (Qwen2.5-Omni, 7B) to a diverse task suite spanning ASR, text and speech summarization, dialect identification, and emotion recognition. It introduces AraMega-SSum, a large Arabic short-speech summarization dataset, and analyzes four training regimes—Stochastic Mixing, Task-Progressive Curriculum, Aligner-Based Diverse Sampling, and a Hybrid of the two—under a fixed compute budget with LoRA. The results reveal a robust efficiency–robustness trade-off: curriculum-style scheduling stabilizes acoustic mapping but can cause negative transfer to higher-level tasks, while diversity-based sampling accelerates early gains but can destabilize generative decoding; a Hybrid strategy mitigates these issues by sequentially building a representative foundation and then refining with diversity. The findings provide practical guidance for efficiently adapting Omni-models to complex, dialect-rich, low-resource multimodal settings and establish AraMega-SSum as a benchmark for Arabic speech understanding and semantic compression.

Abstract

Audio large language models (LLMs) enable unified speech understanding and generation, yet their adaptation to linguistically complex, dialect-rich settings remains underexplored. This paper presents the first systematic study of multi-task instruction tuning for an Arabic-centric audio LLM, covering a hierarchy of generative tasks (ASR, speech summarization) and discriminative tasks (dialect and emotion identification). To support this study, we introduce AraMega-SSum, a novel dataset for Arabic speech summarization. We fine-tune Qwen2.5-Omni (7B) and propose Task-Progressive Curriculum (TPC) along with Aligner-Based Diverse Sampling (ADS), a strategy that constructs information-dense batches by selecting task- and label-balanced examples. Our results reveal a critical efficiency, robustness trade-off: while ADS accelerates initial convergence and boosts paralinguistic F1-scores, its inherent gradient volatility can destabilize generative decoding under prolonged training. Furthermore, while the TPC stabilizes core acoustic mapping, it often induces negative transfer in downstream tasks. We demonstrate that a Hybrid TPC+ADS Strategy provides an optimal training ``recipe'', first establishing a robust representative foundation before employing diversity-aware refinement to capture fine-grained nuances. These findings offer practical guidance for the efficient adaptation of Omni-models in complex, low-resource multimodal environments.

Harmonizing the Arabic Audio Space with Data Scheduling

TL;DR

This work studies how data scheduling affects the adaptation of an Arabic-centric audio LLM (Qwen2.5-Omni, 7B) to a diverse task suite spanning ASR, text and speech summarization, dialect identification, and emotion recognition. It introduces AraMega-SSum, a large Arabic short-speech summarization dataset, and analyzes four training regimes—Stochastic Mixing, Task-Progressive Curriculum, Aligner-Based Diverse Sampling, and a Hybrid of the two—under a fixed compute budget with LoRA. The results reveal a robust efficiency–robustness trade-off: curriculum-style scheduling stabilizes acoustic mapping but can cause negative transfer to higher-level tasks, while diversity-based sampling accelerates early gains but can destabilize generative decoding; a Hybrid strategy mitigates these issues by sequentially building a representative foundation and then refining with diversity. The findings provide practical guidance for efficiently adapting Omni-models to complex, dialect-rich, low-resource multimodal settings and establish AraMega-SSum as a benchmark for Arabic speech understanding and semantic compression.

Abstract

Audio large language models (LLMs) enable unified speech understanding and generation, yet their adaptation to linguistically complex, dialect-rich settings remains underexplored. This paper presents the first systematic study of multi-task instruction tuning for an Arabic-centric audio LLM, covering a hierarchy of generative tasks (ASR, speech summarization) and discriminative tasks (dialect and emotion identification). To support this study, we introduce AraMega-SSum, a novel dataset for Arabic speech summarization. We fine-tune Qwen2.5-Omni (7B) and propose Task-Progressive Curriculum (TPC) along with Aligner-Based Diverse Sampling (ADS), a strategy that constructs information-dense batches by selecting task- and label-balanced examples. Our results reveal a critical efficiency, robustness trade-off: while ADS accelerates initial convergence and boosts paralinguistic F1-scores, its inherent gradient volatility can destabilize generative decoding under prolonged training. Furthermore, while the TPC stabilizes core acoustic mapping, it often induces negative transfer in downstream tasks. We demonstrate that a Hybrid TPC+ADS Strategy provides an optimal training ``recipe'', first establishing a robust representative foundation before employing diversity-aware refinement to capture fine-grained nuances. These findings offer practical guidance for the efficient adaptation of Omni-models in complex, low-resource multimodal environments.
Paper Structure (30 sections, 7 figures, 6 tables, 1 algorithm)

This paper contains 30 sections, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Formalization
  5. Generative tasks.
  6. Classification tasks.
  7. Phase 1: Language-centric Alignment
  8. Phase 2: Multi-task Training Regimes
  9. Datasets
  10. AraMega-SSum: Arabic Short Speech Summarization
  11. Translation and Semantic Alignment.
  12. Audio Synthesis via Neural Voice Cloning.
  13. Training Datasets
  14. Evaluation Benchmarks
  15. Automatic Speech Recognition (ASR).
  16. ...and 15 more sections

Figures (7)

  • Figure 1: Training loss comparison across multi-task training setup.
  • Figure 2: Emotion label distribution across training datasets.
  • Figure 3: Dialect distribution in the training set.
  • Figure 4: Sample speech summarization instances from AraMega-SSum.
  • Figure 5: Prompt used for LLM-as-a-judge summarization evaluation.
  • ...and 2 more figures