An End-to-End Speech Summarization Using Large Language Model

TL;DR

This work tackles end-to-end Abstractive Speech Summarization (SSum) for long inputs by integrating a S-Encoder with a cross-modal Q-Former and a pretrained LLM (LLaMA2-7B) fine-tuned via LoRA. It introduces a three-stage training regime—sentence-level ASR, document-level ASR, and curriculum learning transitioning from Text Summarization (TSum) to SSum—to bridge modality gaps and handle long-context speech. Experiments on How-2 show the proposed model outperforms cascaded ASR+TSum baselines and approaches the performance of GT-transcript baselines on key metrics, with ablations confirming the importance of long-context ASR and curriculum learning. The approach demonstrates the viability of leveraging LLMs for end-to-end SSum, offering insights into cross-modal alignment, long-speech processing, and efficient fine-tuning with LoRA for practical deployment.$\mathcal{L}_{LM}= -\sum_{i=1}^{T_{sum}} \log P(y_i|y_{<i}, F_{speech} \oplus E_{audio}; \theta_{LoRA})$ highlighting the central optimization objective. This work thus advances end-to-end SSum by enabling direct generation of concise textual summaries from speech while mitigating error propagation and modality gaps.

Abstract

Abstractive Speech Summarization (SSum) aims to generate human-like text summaries from spoken content. It encounters difficulties in handling long speech input and capturing the intricate cross-modal mapping between long speech inputs and short text summaries. Research on large language models (LLMs) and multimodal information fusion has provided new insights for addressing these challenges. In this paper, we propose an end-to-end SSum model that utilizes Q-Former as a connector for the audio-text modality and employs LLMs to generate text summaries directly from speech features. We adopt a multi-stage training approach that includes LLM based ASR and Text Summarization (TSum) tasks as auxiliary tasks. ASR tasks are used to align feature spaces and enhance the LLM's ability to handle longer speech. Then, we utilize a curriculum learning strategy to facilitate the model's transition from TSum to SSum. Finally, our model achieves competitive performance on the How-2 dataset.

Figures (3)

• Figure 1: Overview of the proposed model. A speech encoder and Q-Former are used to extract speech features. LLM accepts speech prompts and generates text summaries directly. Text transcripts are used as auxiliary information during the training.
• Figure 2: The training pipline of our model.
• Figure 3: The weight distribution in the weighted sum module of different models obtained from different training tasks.