Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Abstractive summarization from Audio Transcription

Ilia Derkach

TL;DR

This paper tackles the resource-intensive challenge of fine-tuning large language models for end-to-end audio summarization by advocating parameter-efficient fine-tuning (PEFT) and quantization. It evaluates MBart and T5 as text-to-text summarizers and Whisper as the ASR component, applying LoRA, AdaLoRA, and QLoRA to measure efficiency and accuracy on Gazeta and WikiHow datasets using ROUGE and WER metrics. The results show AdaLoRA generally outperforms LoRA and approaches full fine-tuning performance while delivering substantial reductions in training time and memory, and quantization further enhances efficiency, enabling practical end-to-end S2T summarization. The work highlights a viable path to deploy efficient E2E audio summarization systems and outlines future directions, including stronger compression, additional modalities like video, and ranking-based improvements to summarization quality.

Abstract

Currently, large language models are gaining popularity, their achievements are used in many areas, ranging from text translation to generating answers to queries. However, the main problem with these new machine learning algorithms is that training such models requires large computing resources that only large IT companies have. To avoid this problem, a number of methods (LoRA, quantization) have been proposed so that existing models can be effectively fine-tuned for specific tasks. In this paper, we propose an E2E (end to end) audio summarization model using these techniques. In addition, this paper examines the effectiveness of these approaches to the problem under consideration and draws conclusions about the applicability of these methods.

Abstractive summarization from Audio Transcription

TL;DR

This paper tackles the resource-intensive challenge of fine-tuning large language models for end-to-end audio summarization by advocating parameter-efficient fine-tuning (PEFT) and quantization. It evaluates MBart and T5 as text-to-text summarizers and Whisper as the ASR component, applying LoRA, AdaLoRA, and QLoRA to measure efficiency and accuracy on Gazeta and WikiHow datasets using ROUGE and WER metrics. The results show AdaLoRA generally outperforms LoRA and approaches full fine-tuning performance while delivering substantial reductions in training time and memory, and quantization further enhances efficiency, enabling practical end-to-end S2T summarization. The work highlights a viable path to deploy efficient E2E audio summarization systems and outlines future directions, including stronger compression, additional modalities like video, and ranking-based improvements to summarization quality.

Abstract

Currently, large language models are gaining popularity, their achievements are used in many areas, ranging from text translation to generating answers to queries. However, the main problem with these new machine learning algorithms is that training such models requires large computing resources that only large IT companies have. To avoid this problem, a number of methods (LoRA, quantization) have been proposed so that existing models can be effectively fine-tuned for specific tasks. In this paper, we propose an E2E (end to end) audio summarization model using these techniques. In addition, this paper examines the effectiveness of these approaches to the problem under consideration and draws conclusions about the applicability of these methods.
Paper Structure (19 sections, 18 equations, 6 figures, 8 tables)

This paper contains 19 sections, 18 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning
  3. Low-Rank Adaptation (LoRA)
  4. Adaptive budget allocation for parameter-efficient fine-tuning (AdaLoRA)
  5. Quantization
  6. Efficient Finetuning of Quantized LLMs (QLoRA)
  7. Text-to-text summarization
  8. MBart
  9. T5
  10. Summarization metrics
  11. Dataset for text summarization
  12. Fine-tuning text summarizarion models
  13. Automatic Speech Recognition
  14. Whisper
  15. Dataset and metric
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Schematic representation of LoRA. Weights $W$ are frozen, while $\Delta W = A\cdot B$
  • Figure 2: Training plot for fine-tuning MBart and T5 models with LoRA with different $r$
  • Figure 3: ROUGE-1 metric for fine-tuning MBart and T5 models with adapters with different $r$ with Gazeta dataset
  • Figure 4: ROUGE-1 metric for fine-tuning MBart and T5 models with adapters with different $r$ with WikiHow dataset
  • Figure 5: WER during fine-tuning Whisper model with different amount of weights
  • ...and 1 more figures