Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Cascaded Architecture for Extractive Summarization of Multimedia Content via Audio-to-Text Alignment

Tanzir Hossain, Ar-Rafi Islam, Md. Sabbir Hossain, Annajiat Alim Rasel

TL;DR

This work tackles the challenge of extracting concise summaries from multimedia content by proposing a cascaded architecture that first aligns audio with text transcripts and then applies a combination of extractive and abstractive summarization. The pipeline leverages audio-to-text transcription (Azure Speech/Whisper and Google ASR), frequency-based sentence scoring, a LEAD-based extractor, and BART-Large for abstractive fusion, with training anchored on the XSum dataset. Experiments reveal competitive performance with traditional methods on ROUGE metrics, but also reveal coherence and fluency limitations, partly due to undertraining and data constraints. Overall, the study demonstrates the feasibility of end-to-end multimedia summarization leveraging audio alignment to unlock textual NLP techniques, with significant implications for information retrieval, accessibility, and scalable content distillation across platforms.

Abstract

This study presents a cascaded architecture for extractive summarization of multimedia content via audio-to-text alignment. The proposed framework addresses the challenge of extracting key insights from multimedia sources like YouTube videos. It integrates audio-to-text conversion using Microsoft Azure Speech with advanced extractive summarization models, including Whisper, Pegasus, and Facebook BART XSum. The system employs tools such as Pytube, Pydub, and SpeechRecognition for content retrieval, audio extraction, and transcription. Linguistic analysis is enhanced through named entity recognition and semantic role labeling. Evaluation using ROUGE and F1 scores demonstrates that the cascaded architecture outperforms conventional summarization methods, despite challenges like transcription errors. Future improvements may include model fine-tuning and real-time processing. This study contributes to multimedia summarization by improving information retrieval, accessibility, and user experience.

A Cascaded Architecture for Extractive Summarization of Multimedia Content via Audio-to-Text Alignment

TL;DR

This work tackles the challenge of extracting concise summaries from multimedia content by proposing a cascaded architecture that first aligns audio with text transcripts and then applies a combination of extractive and abstractive summarization. The pipeline leverages audio-to-text transcription (Azure Speech/Whisper and Google ASR), frequency-based sentence scoring, a LEAD-based extractor, and BART-Large for abstractive fusion, with training anchored on the XSum dataset. Experiments reveal competitive performance with traditional methods on ROUGE metrics, but also reveal coherence and fluency limitations, partly due to undertraining and data constraints. Overall, the study demonstrates the feasibility of end-to-end multimedia summarization leveraging audio alignment to unlock textual NLP techniques, with significant implications for information retrieval, accessibility, and scalable content distillation across platforms.

Abstract

This study presents a cascaded architecture for extractive summarization of multimedia content via audio-to-text alignment. The proposed framework addresses the challenge of extracting key insights from multimedia sources like YouTube videos. It integrates audio-to-text conversion using Microsoft Azure Speech with advanced extractive summarization models, including Whisper, Pegasus, and Facebook BART XSum. The system employs tools such as Pytube, Pydub, and SpeechRecognition for content retrieval, audio extraction, and transcription. Linguistic analysis is enhanced through named entity recognition and semantic role labeling. Evaluation using ROUGE and F1 scores demonstrates that the cascaded architecture outperforms conventional summarization methods, despite challenges like transcription errors. Future improvements may include model fine-tuning and real-time processing. This study contributes to multimedia summarization by improving information retrieval, accessibility, and user experience.

Paper Structure

This paper contains 19 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Methodology
  4. Data Collection:
  5. Dataset Analysis:
  6. Speech-to-Text Alignment
  7. Text Preprocessing:
  8. Extractive Summarization Pipeline:
  9. Evaluation Metrics:
  10. Experimental Setup and Result Analysis
  11. Data:
  12. Preprocessing:
  13. Model Architecture:
  14. Training Details:
  15. Evaluation Metrics:
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Xsum Data