Cross-Lingual Conversational Speech Summarization with Large Language Models

TL;DR

This work tackles cross-lingual conversational speech summarization under limited resources by building a public CTS dataset from Fisher and Callhome Spanish–English corpora and generating ground-truth summaries with GPT-4 from reference translations. It evaluates a cascaded ASR/MT/LLM pipeline across multiple models, revealing that a LoRA-fine-tuned Mistral-7B can match GPT-4 performance, demonstrating the viability of smaller, adaptable LLMs for this task. The study also analyzes robustness to transcription and translation errors, finding only modest degradation in summarization quality, and establishes a practical evaluation framework for cross-lingual CTS summarization with potential extensions to contextual and user-guided summaries.

Abstract

Cross-lingual conversational speech summarization is an important problem, but suffers from a dearth of resources. While transcriptions exist for a number of languages, translated conversational speech is rare and datasets containing summaries are non-existent. We build upon the existing Fisher and Callhome Spanish-English Speech Translation corpus by supplementing the translations with summaries. The summaries are generated using GPT-4 from the reference translations and are treated as ground truth. The task is to generate similar summaries in the presence of transcription and translation errors. We build a baseline cascade-based system using open-source speech recognition and machine translation models. We test a range of LLMs for summarization and analyze the impact of transcription and translation errors. Adapting the Mistral-7B model for this task performs significantly better than off-the-shelf models and matches the performance of GPT-4.